[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:56 dist-test-slave-v7xz krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:56 dist-test-slave-v7xz krb5kdc[399](info): set up 2 sockets
Sep 28 21:11:56 dist-test-slave-v7xz krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260928 21:11:57.229301 392 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:11:57 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:11:57.348620 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43481
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43481
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:57.456511 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:57.456820 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:57.456918 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:57.460894 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:57.460947 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:57.460968 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:57.460983 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:57.460999 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:57.466146 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43481
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:57.467079 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:57.468080 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:57.476787 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:57.476943 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:57.476938 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:57.477077 412 server_base.cc:1034] running on GCE node
I20260928 21:11:57.477648 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:57.478665 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:57.479854 412 hybrid_clock.cc:648] HybridClock initialized: now 1790629917479810 us; error 57 us; skew 500 ppm
Sep 28 21:11:57 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:57.482565 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:11:57.483359 412 webserver.cc:460] Webserver started at http://127.0.98.62:38493/ using document root <none> and password file <none>
I20260928 21:11:57.483925 412 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:57.483986 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:57.484148 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:57.486157 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "eda6fd98701b458ba1270f3efb94a683"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "0ecbca17d32618b0919f6a30d95e5598"
server_key_iv: "ad787f07ebd03921cad3eb2fe73262d9"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.486879 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "eda6fd98701b458ba1270f3efb94a683"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "0ecbca17d32618b0919f6a30d95e5598"
server_key_iv: "ad787f07ebd03921cad3eb2fe73262d9"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.490715 412 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:11:57.493062 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:57.494308 412 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:57.494431 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "eda6fd98701b458ba1270f3efb94a683"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "0ecbca17d32618b0919f6a30d95e5598"
server_key_iv: "ad787f07ebd03921cad3eb2fe73262d9"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.494534 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:57.515604 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:57.518416 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:57.518635 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:57.526006 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43481
I20260928 21:11:57.526039 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43481 every 8 connection(s)
I20260928 21:11:57.526929 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:57.529625 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:57.534826 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683: Bootstrap starting.
I20260928 21:11:57.535005 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 412
I20260928 21:11:57.535169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:11:57.536446 392 external_mini_cluster.cc:1428] Setting key 24e1e03df90c329abbb5401af3747fb2
I20260928 21:11:57.537420 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:57.538128 480 log.cc:824] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:57.540343 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683: No bootstrap required, opened a new log
I20260928 21:11:57.544171 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda6fd98701b458ba1270f3efb94a683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43481 } }
I20260928 21:11:57.544412 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:57.544507 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eda6fd98701b458ba1270f3efb94a683, State: Initialized, Role: FOLLOWER
I20260928 21:11:57.545012 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda6fd98701b458ba1270f3efb94a683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43481 } }
I20260928 21:11:57.545168 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:57.545255 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:57.545377 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:57.546310 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda6fd98701b458ba1270f3efb94a683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43481 } }
I20260928 21:11:57.546649 480 leader_election.cc:302] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eda6fd98701b458ba1270f3efb94a683; no voters:
I20260928 21:11:57.546983 480 leader_election.cc:288] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:57.547079 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:57.547291 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [term 1 LEADER]: Becoming Leader. State: Replica: eda6fd98701b458ba1270f3efb94a683, State: Running, Role: LEADER
I20260928 21:11:57.547617 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda6fd98701b458ba1270f3efb94a683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43481 } }
I20260928 21:11:57.548213 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:57.549295 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eda6fd98701b458ba1270f3efb94a683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda6fd98701b458ba1270f3efb94a683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43481 } } }
I20260928 21:11:57.549412 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:57.549759 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eda6fd98701b458ba1270f3efb94a683. Latest consensus state: current_term: 1 leader_uuid: "eda6fd98701b458ba1270f3efb94a683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eda6fd98701b458ba1270f3efb94a683" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43481 } } }
I20260928 21:11:57.550594 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683 [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:57.550930 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:57.553670 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 28 21:11:57 dist-test-slave-v7xz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:11:57.559541 494 catalog_manager.cc:1347] Generated new cluster ID: d4ba26510a3e407092579664dab8127e
I20260928 21:11:57.559659 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:57.561546 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:57.539423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.0.98.62:43481)
0928 21:11:57.539811 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:57.539833 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:57.541117 (+ 1284us) server_negotiation.cc:407] Connection header received
0928 21:11:57.543803 (+ 2686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:57.543817 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:57.544107 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:57.544432 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:57.546849 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:57.548104 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:57.549801 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:57.550504 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:57.550684 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:57.555469 (+ 4785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:57.555476 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:57.555494 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:57.555528 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:57.557826 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:57.558420 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:57.558425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:57.558428 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:57.558559 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:57.558905 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:57.558914 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:57.558916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:57.559387 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:57.559617 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:57.560387 (+ 770us) server_negotiation.cc:299] Negotiation successful
0928 21:11:57.560672 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":143,"threads_started":1}
I20260928 21:11:57.567776 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:57.568639 494 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:57.575424 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eda6fd98701b458ba1270f3efb94a683: Generated new TSK 0
I20260928 21:11:57.576077 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:11:57.607690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43481
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:57.738770 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:57.739010 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:57.739054 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:57.743008 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:57.743065 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:57.743124 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:11:57.748232 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:57.749307 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:57.750342 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:57.759941 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:57.759982 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:57.759946 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:57.760277 506 server_base.cc:1034] running on GCE node
I20260928 21:11:57.760707 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:57.761399 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:57.762588 506 hybrid_clock.cc:648] HybridClock initialized: now 1790629917762574 us; error 32 us; skew 500 ppm
Sep 28 21:11:57 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:57.765777 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:11:57.766712 506 webserver.cc:460] Webserver started at http://127.0.98.1:33843/ using document root <none> and password file <none>
I20260928 21:11:57.767280 506 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:57.767369 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:57.767601 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:57.769727 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "b1dde8fbd7e7479aa096e045e32eef02"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "fd349a96b572a388117ad4410b9e35de"
server_key_iv: "0fe4b08731c1b342c13106be030be7db"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.770298 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b1dde8fbd7e7479aa096e045e32eef02"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "fd349a96b572a388117ad4410b9e35de"
server_key_iv: "0fe4b08731c1b342c13106be030be7db"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.773967 506 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 21:11:57.776371 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:57.777726 506 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:57.777875 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "b1dde8fbd7e7479aa096e045e32eef02"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "fd349a96b572a388117ad4410b9e35de"
server_key_iv: "0fe4b08731c1b342c13106be030be7db"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.777994 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:57.803056 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:57.806286 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:57.806492 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:57.807078 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:11:57.807900 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:11:57.807973 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:57.808039 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:11:57.808084 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:57.816922 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39115
I20260928 21:11:57.816942 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39115 every 8 connection(s)
I20260928 21:11:57.817824 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:11:57.824891 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 506
I20260928 21:11:57.825008 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:11:57.825309 392 external_mini_cluster.cc:1428] Setting key d71eb0bc9f5889a23b50fe6b21b41ff4
Sep 28 21:11:57 dist-test-slave-v7xz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:11:57.831877 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:57.819596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37183 (local address 127.0.98.62:43481)
0928 21:11:57.819750 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:57.819753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:57.820519 (+ 766us) server_negotiation.cc:407] Connection header received
0928 21:11:57.821872 (+ 1353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:57.821876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:57.821932 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:57.822057 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:57.823201 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:57.823736 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:57.825182 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:57.825548 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:57.825675 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:57.828257 (+ 2582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:57.828260 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:57.828263 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:57.828294 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:57.829810 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:57.830343 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:57.830347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:57.830349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:57.830428 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:57.830773 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:57.830776 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:57.830778 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:57.830977 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:57.831084 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:57.831585 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 21:11:57.831709 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:11:57.832671 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:57.819928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43481 (local address 127.0.98.1:37183)
0928 21:11:57.820362 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:57.820403 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:57.821660 (+ 1257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:57.822230 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:57.822237 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:57.822590 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:57.823060 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:57.823073 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:57.823860 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:57.823864 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:57.825053 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:57.825064 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:57.825698 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:57.825701 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:57.825909 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:57.826577 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:57.826603 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:57.828094 (+ 1491us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:57.829956 (+ 1862us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:57.829963 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:57.829976 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:57.830240 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:57.830525 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:57.830527 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:57.830529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:57.830663 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:57.831082 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:57.831089 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:57.831343 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 21:11:57.831534 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 21:11:57.831848 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:11:57.834079 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43481
I20260928 21:11:57.834316 635 heartbeater.cc:464] Registering TS with master...
I20260928 21:11:57.834832 635 heartbeater.cc:511] Master 127.0.98.62:43481 requested a full tablet report, sending...
I20260928 21:11:57.836324 441 ts_manager.cc:194] Registered new tserver with Master: b1dde8fbd7e7479aa096e045e32eef02 (127.0.98.1:39115)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:11:57.837437 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:11:57.851104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43481
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:57.960880 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:57.961164 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:57.961211 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:57.965327 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:57.965387 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:57.965464 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:11:57.970839 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:57.971874 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:57.972924 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:57.982491 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:57.982491 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:57.982493 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:57.983194 640 server_base.cc:1034] running on GCE node
I20260928 21:11:57.983563 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:57.984189 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:57.985368 640 hybrid_clock.cc:648] HybridClock initialized: now 1790629917985349 us; error 41 us; skew 500 ppm
Sep 28 21:11:57 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:57.988178 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:11:57.989180 640 webserver.cc:460] Webserver started at http://127.0.98.2:33731/ using document root <none> and password file <none>
I20260928 21:11:57.989764 640 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:57.989858 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:57.990090 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:57.992141 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "b595f41d60de46b899c0f129ce302efc"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "b4ba67785c288424b0a983cbf17edfd0"
server_key_iv: "673bd30a64dbd39b03a567fc9cca0146"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.992712 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b595f41d60de46b899c0f129ce302efc"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "b4ba67785c288424b0a983cbf17edfd0"
server_key_iv: "673bd30a64dbd39b03a567fc9cca0146"
server_key_version: "encryptionkey@0"
I20260928 21:11:57.996361 640 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:11:57.998848 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:58.000079 640 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:58.000209 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "b595f41d60de46b899c0f129ce302efc"
format_stamp: "Formatted at 2026-09-28 21:11:57 on dist-test-slave-v7xz"
server_key: "b4ba67785c288424b0a983cbf17edfd0"
server_key_iv: "673bd30a64dbd39b03a567fc9cca0146"
server_key_version: "encryptionkey@0"
I20260928 21:11:58.000327 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:58.025483 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:58.029052 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:58.029271 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:58.029891 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:11:58.030741 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:11:58.030813 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:58.030882 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:11:58.030930 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:58.041299 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39947
I20260928 21:11:58.041320 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39947 every 8 connection(s)
I20260928 21:11:58.042250 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:11:58.047835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 640
I20260928 21:11:58.047935 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:11:58.048261 392 external_mini_cluster.cc:1428] Setting key 9e904d527602ae0e9a83a9e1db54f5fa
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:11:58.057186 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:58.044198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44781 (local address 127.0.98.62:43481)
0928 21:11:58.044347 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:58.044352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:58.045331 (+ 979us) server_negotiation.cc:407] Connection header received
0928 21:11:58.046669 (+ 1338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:58.046672 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:58.046723 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:58.046803 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:58.048032 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.048861 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.050271 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.050523 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.050671 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:58.053374 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:58.053377 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:58.053380 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:58.053404 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:58.054907 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.055451 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.055464 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.055465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.055546 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.055933 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.055936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.055938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.056191 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:58.056315 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:58.056824 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 21:11:58.057013 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:11:58.057999 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:58.044668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43481 (local address 127.0.98.2:44781)
0928 21:11:58.045174 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:58.045220 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:58.046461 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:58.046975 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:58.046982 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:58.047359 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:58.047839 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:58.047851 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.049022 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.049025 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:58.050151 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:58.050160 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.050704 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.050708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:58.050891 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:58.051560 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:58.051577 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:58.053199 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:58.055047 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:58.055054 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:58.055070 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:58.055342 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:58.055646 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:58.055649 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:58.055651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:58.055796 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:58.056313 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:58.056317 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:58.056570 (+ 253us) client_negotiation.cc:769] Sending connection context
0928 21:11:58.056730 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 21:11:58.057080 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":164,"threads_started":1}
I20260928 21:11:58.059383 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43481
I20260928 21:11:58.059689 769 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:11:58.060362 769 heartbeater.cc:511] Master 127.0.98.62:43481 requested a full tablet report, sending...
I20260928 21:11:58.061563 441 ts_manager.cc:194] Registered new tserver with Master: b595f41d60de46b899c0f129ce302efc (127.0.98.2:39947)
I20260928 21:11:58.062156 441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44781
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:11:58.077754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43481
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:58.191708 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:58.192092 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:58.192179 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:58.197144 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:58.197204 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:58.197266 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:11:58.202728 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43481
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:58.203728 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:58.204757 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:58.214298 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:58.214305 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:58.214305 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:58.214500 774 server_base.cc:1034] running on GCE node
I20260928 21:11:58.214951 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:58.215617 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:58.216799 774 hybrid_clock.cc:648] HybridClock initialized: now 1790629918216765 us; error 51 us; skew 500 ppm
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:58.219532 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:11:58.220410 774 webserver.cc:460] Webserver started at http://127.0.98.3:35015/ using document root <none> and password file <none>
I20260928 21:11:58.221058 774 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:58.221148 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:58.221374 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:58.223394 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "b5e20acdc74c4d5a9c4cbfe7d977fd3d"
format_stamp: "Formatted at 2026-09-28 21:11:58 on dist-test-slave-v7xz"
server_key: "c539ffd06ae7c6b55722db3c3f5019c7"
server_key_iv: "1e8cf62c6705cf378471280f34f17de8"
server_key_version: "encryptionkey@0"
I20260928 21:11:58.223966 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "b5e20acdc74c4d5a9c4cbfe7d977fd3d"
format_stamp: "Formatted at 2026-09-28 21:11:58 on dist-test-slave-v7xz"
server_key: "c539ffd06ae7c6b55722db3c3f5019c7"
server_key_iv: "1e8cf62c6705cf378471280f34f17de8"
server_key_version: "encryptionkey@0"
I20260928 21:11:58.227700 774 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:11:58.230123 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:58.231292 774 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:11:58.231424 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "b5e20acdc74c4d5a9c4cbfe7d977fd3d"
format_stamp: "Formatted at 2026-09-28 21:11:58 on dist-test-slave-v7xz"
server_key: "c539ffd06ae7c6b55722db3c3f5019c7"
server_key_iv: "1e8cf62c6705cf378471280f34f17de8"
server_key_version: "encryptionkey@0"
I20260928 21:11:58.231568 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:58.244941 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:58.247987 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:58.248239 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:58.249023 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:11:58.250064 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:11:58.250128 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:58.250191 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:11:58.250238 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:58.259780 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39179
I20260928 21:11:58.259805 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39179 every 8 connection(s)
I20260928 21:11:58.260722 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:11:58.263770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 774
I20260928 21:11:58.263893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:11:58.264214 392 external_mini_cluster.cc:1428] Setting key ef13d5fa40cdec9f7d08f116157a33ed
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:11:58.276141 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:58.262647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34247 (local address 127.0.98.62:43481)
0928 21:11:58.262805 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:58.262809 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:58.263788 (+ 979us) server_negotiation.cc:407] Connection header received
0928 21:11:58.265450 (+ 1662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:58.265454 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:58.265512 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:58.265622 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:58.266806 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.267336 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.268496 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.268725 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.268839 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:58.272011 (+ 3172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:58.272014 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:58.272016 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:58.272042 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:58.273876 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.274417 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.274420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.274422 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.274508 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.274900 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.274902 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.274903 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.275101 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:58.275191 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:58.275725 (+ 534us) server_negotiation.cc:299] Negotiation successful
0928 21:11:58.275926 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:11:58.276919 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:58.263065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43481 (local address 127.0.98.3:34247)
0928 21:11:58.263584 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0928 21:11:58.263647 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:11:58.265220 (+ 1573us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:11:58.265807 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:11:58.265813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:11:58.266174 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:11:58.266650 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:58.266674 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.267468 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.267478 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:58.268368 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:11:58.268376 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.268905 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.268909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:11:58.269146 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:58.270149 (+ 1003us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:11:58.270166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:11:58.271834 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:11:58.273991 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:58.273996 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:58.274010 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:58.274294 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:58.274635 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:11:58.274637 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:11:58.274638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:11:58.274806 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:11:58.275201 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:11:58.275207 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:11:58.275466 (+ 259us) client_negotiation.cc:769] Sending connection context
0928 21:11:58.275628 (+ 162us) client_negotiation.cc:240] Negotiation successful
0928 21:11:58.275928 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260928 21:11:58.278409 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43481
I20260928 21:11:58.278729 903 heartbeater.cc:464] Registering TS with master...
I20260928 21:11:58.279328 903 heartbeater.cc:511] Master 127.0.98.62:43481 requested a full tablet report, sending...
I20260928 21:11:58.280407 444 ts_manager.cc:194] Registered new tserver with Master: b5e20acdc74c4d5a9c4cbfe7d977fd3d (127.0.98.3:39179)
I20260928 21:11:58.281090 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34247
I20260928 21:11:58.291428 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629918, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:11:58.318750 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:58.309008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.0.98.1:39115)
0928 21:11:58.309335 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:58.309350 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:58.309378 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:11:58.309473 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:58.309480 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:58.309697 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:58.309859 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:58.310571 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.311343 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.312196 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.312582 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.312726 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:58.315179 (+ 2453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:58.315187 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:58.315201 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:58.315225 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:58.316815 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.317311 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.317324 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.317326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.317407 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.317690 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.317694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.317695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.317978 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:58.318222 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:58.318401 (+ 179us) server_negotiation.cc:299] Negotiation successful
0928 21:11:58.318558 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":96,"threads_started":1}
W20260928 21:11:58.319814 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46608
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629918, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:11:58.346632 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:58.337388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46614 (local address 127.0.98.1:39115)
0928 21:11:58.337567 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:58.337572 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:58.337704 (+ 132us) server_negotiation.cc:407] Connection header received
0928 21:11:58.337892 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:58.337896 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:58.337950 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:58.338051 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:58.338796 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.339365 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.340296 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:58.340575 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:58.340703 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:58.343449 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:58.343453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:58.343455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:58.343480 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:58.345012 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.345445 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.345448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.345449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.345583 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:58.345893 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:58.345895 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:58.345897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:58.346086 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:58.346187 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:58.346405 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 21:11:58.346510 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:11:58.347754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 506
I20260928 21:11:58.353904 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 640
I20260928 21:11:58.361662 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 774
I20260928 21:11:58.367223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 412
2026-09-28T21:11:58Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2239 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[917](info): set up 2 sockets
Sep 28 21:11:58 dist-test-slave-v7xz krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260928 21:11:59.404621 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:11:59 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:11:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:11:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:11:59.523904 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43447
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43447
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:59.649696 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:59.649977 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:59.650019 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:59.654047 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:11:59.654122 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:59.654143 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:59.654160 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:11:59.654175 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:11:59.659570 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43447
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:59.660653 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:59.661667 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:59.670233 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:59.670281 930 server_base.cc:1034] running on GCE node
W20260928 21:11:59.670439 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:59.670239 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:59.671188 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:59.672338 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:59.673533 930 hybrid_clock.cc:648] HybridClock initialized: now 1790629919673497 us; error 50 us; skew 500 ppm
Sep 28 21:11:59 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:59.676373 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:11:59.677405 930 webserver.cc:460] Webserver started at http://127.0.98.62:36343/ using document root <none> and password file <none>
I20260928 21:11:59.677984 930 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:59.678079 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:59.678315 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:59.680344 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "e32c12c308d3400e9a40c5a551d9559f"
format_stamp: "Formatted at 2026-09-28 21:11:59 on dist-test-slave-v7xz"
server_key: "aa5fad9eb46d0481129b21957e935d5c"
server_key_iv: "1a7f05d6f58b3f8529c90baedf33924c"
server_key_version: "encryptionkey@0"
I20260928 21:11:59.680953 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "e32c12c308d3400e9a40c5a551d9559f"
format_stamp: "Formatted at 2026-09-28 21:11:59 on dist-test-slave-v7xz"
server_key: "aa5fad9eb46d0481129b21957e935d5c"
server_key_iv: "1a7f05d6f58b3f8529c90baedf33924c"
server_key_version: "encryptionkey@0"
I20260928 21:11:59.684584 930 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:11:59.686971 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:59.688102 930 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:11:59.688256 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "e32c12c308d3400e9a40c5a551d9559f"
format_stamp: "Formatted at 2026-09-28 21:11:59 on dist-test-slave-v7xz"
server_key: "aa5fad9eb46d0481129b21957e935d5c"
server_key_iv: "1a7f05d6f58b3f8529c90baedf33924c"
server_key_version: "encryptionkey@0"
I20260928 21:11:59.688362 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:59.706130 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:59.709628 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:59.709848 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:59.716971 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43447
I20260928 21:11:59.716997 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43447 every 8 connection(s)
I20260928 21:11:59.717957 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:11:59.719732 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 930
I20260928 21:11:59.719846 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:11:59.720094 392 external_mini_cluster.cc:1428] Setting key 807587b49e472eab38b10bbf54b97776
I20260928 21:11:59.721074 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:11:59.727514 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f: Bootstrap starting.
Sep 28 21:11:59 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:11:59.729987 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f: Neither blocks nor log segments found. Creating new log.
I20260928 21:11:59.730794 998 log.cc:824] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:11:59.732687 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f: No bootstrap required, opened a new log
I20260928 21:11:59.734671 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:11:59.721470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40510 (local address 127.0.98.62:43447)
0928 21:11:59.721875 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 21:11:59.721893 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:11:59.721937 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:11:59.722554 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:11:59.722570 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:11:59.722915 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:11:59.723205 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:11:59.724106 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:59.725193 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:59.726429 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:11:59.726948 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:11:59.727081 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:11:59.729976 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:11:59.729983 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:11:59.729998 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:11:59.730024 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:11:59.731793 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:59.732245 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:59.732251 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:59.732257 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:59.732375 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:11:59.732665 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:11:59.732670 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:11:59.732671 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:11:59.733166 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:11:59.733369 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0928 21:11:59.733671 (+ 302us) server_negotiation.cc:299] Negotiation successful
0928 21:11:59.733963 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":152,"threads_started":1}
I20260928 21:11:59.736634 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c12c308d3400e9a40c5a551d9559f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260928 21:11:59.736953 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:11:59.737025 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32c12c308d3400e9a40c5a551d9559f, State: Initialized, Role: FOLLOWER
I20260928 21:11:59.737491 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c12c308d3400e9a40c5a551d9559f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260928 21:11:59.737619 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:11:59.737659 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:11:59.737730 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:11:59.738731 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c12c308d3400e9a40c5a551d9559f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260928 21:11:59.739099 998 leader_election.cc:302] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32c12c308d3400e9a40c5a551d9559f; no voters:
I20260928 21:11:59.739372 998 leader_election.cc:288] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:11:59.739661 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:11:59.739866 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [term 1 LEADER]: Becoming Leader. State: Replica: e32c12c308d3400e9a40c5a551d9559f, State: Running, Role: LEADER
I20260928 21:11:59.740185 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c12c308d3400e9a40c5a551d9559f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } }
I20260928 21:11:59.740579 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:11:59.741864 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e32c12c308d3400e9a40c5a551d9559f. Latest consensus state: current_term: 1 leader_uuid: "e32c12c308d3400e9a40c5a551d9559f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c12c308d3400e9a40c5a551d9559f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } } }
I20260928 21:11:59.742009 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [sys.catalog]: This master's current role is: LEADER
I20260928 21:11:59.742301 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32c12c308d3400e9a40c5a551d9559f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32c12c308d3400e9a40c5a551d9559f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43447 } } }
I20260928 21:11:59.742410 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f [sys.catalog]: This master's current role is: LEADER
W20260928 21:11:59.745364 1018 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:11:59.745425 1018 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:11:59.745535 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:11:59.746425 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:11:59.751870 1013 catalog_manager.cc:1347] Generated new cluster ID: 485b2695099b479c8b8bf591a88eecc3
I20260928 21:11:59.751942 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:11:59.781939 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:11:59.782925 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:11:59.792464 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e32c12c308d3400e9a40c5a551d9559f: Generated new TSK 0
I20260928 21:11:59.793426 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:11:59.821758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:11:59.934571 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:11:59.934865 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:11:59.934912 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:11:59.939150 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:11:59.939225 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:11:59.939316 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:11:59.945159 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:11:59.946328 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:11:59.947397 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:11:59.957232 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:59.957261 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:11:59.957261 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:11:59.957572 1024 server_base.cc:1034] running on GCE node
I20260928 21:11:59.958047 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:11:59.958659 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:11:59.959875 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790629919959827 us; error 62 us; skew 500 ppm
Sep 28 21:11:59 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:11:59.962881 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:11:59.963853 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36647/ using document root <none> and password file <none>
I20260928 21:11:59.964465 1024 fs_manager.cc:359] Metadata directory not provided
I20260928 21:11:59.964569 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:11:59.964807 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:11:59.966949 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f71868794dd4dbf83c6f81b38598204"
format_stamp: "Formatted at 2026-09-28 21:11:59 on dist-test-slave-v7xz"
server_key: "34d51ac3b1d46ec92c0138cf5ea9fdea"
server_key_iv: "1e453336a57947aa7e0cf1ceaf26b1ef"
server_key_version: "encryptionkey@0"
I20260928 21:11:59.967535 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f71868794dd4dbf83c6f81b38598204"
format_stamp: "Formatted at 2026-09-28 21:11:59 on dist-test-slave-v7xz"
server_key: "34d51ac3b1d46ec92c0138cf5ea9fdea"
server_key_iv: "1e453336a57947aa7e0cf1ceaf26b1ef"
server_key_version: "encryptionkey@0"
I20260928 21:11:59.971591 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:11:59.974241 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:59.975611 1024 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:11:59.975765 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "8f71868794dd4dbf83c6f81b38598204"
format_stamp: "Formatted at 2026-09-28 21:11:59 on dist-test-slave-v7xz"
server_key: "34d51ac3b1d46ec92c0138cf5ea9fdea"
server_key_iv: "1e453336a57947aa7e0cf1ceaf26b1ef"
server_key_version: "encryptionkey@0"
I20260928 21:11:59.975889 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:11:59.991101 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:11:59.994254 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:11:59.994462 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:11:59.995045 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:11:59.995848 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:11:59.995919 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:11:59.995986 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:11:59.996038 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.004583 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39957
I20260928 21:12:00.004592 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39957 every 8 connection(s)
I20260928 21:12:00.005718 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:00.008229 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1024
I20260928 21:12:00.008332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:00.008594 392 external_mini_cluster.cc:1428] Setting key 1eff30e99bfe44e3062b12e57483d7c0
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:12:00.019979 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.007472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43765 (local address 127.0.98.62:43447)
0928 21:12:00.007634 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.007637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.008459 (+ 822us) server_negotiation.cc:407] Connection header received
0928 21:12:00.010069 (+ 1610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.010072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.010121 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.010192 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.011318 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.011848 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.013043 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.013472 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.013597 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.016129 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.016134 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.016137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.016169 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.017976 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.018478 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.018483 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.018486 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.018595 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.018955 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.018958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.018959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.019145 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.019229 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.019751 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.019856 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:00.020989 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.007811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43447 (local address 127.0.98.1:43765)
0928 21:12:00.008282 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.008362 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.009839 (+ 1477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.010360 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.010367 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.010715 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:00.011178 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.011190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.011973 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.011976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.012928 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.012937 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.013639 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.013642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.013795 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.014476 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:00.014493 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:00.015956 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:00.018094 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:00.018099 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:00.018110 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:00.018376 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:00.018705 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:00.018709 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:00.018711 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:00.018857 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:00.019238 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:00.019243 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:00.019477 (+ 234us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.019706 (+ 229us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.019972 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":154,"threads_started":1}
I20260928 21:12:00.022235 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43447
I20260928 21:12:00.022475 1153 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:00.022976 1153 heartbeater.cc:511] Master 127.0.98.62:43447 requested a full tablet report, sending...
I20260928 21:12:00.024444 962 ts_manager.cc:194] Registered new tserver with Master: 8f71868794dd4dbf83c6f81b38598204 (127.0.98.1:39957)
I20260928 21:12:00.025720 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43765
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:00.036022 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:00.147079 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:00.147398 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:00.147485 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:00.151575 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:00.151674 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:00.151772 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:00.157364 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:00.158470 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:00.159562 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:00.169227 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:00.169229 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:00.169229 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:00.169292 1158 server_base.cc:1034] running on GCE node
I20260928 21:12:00.169909 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:00.170557 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:00.171720 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790629920171687 us; error 47 us; skew 500 ppm
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:00.174546 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:00.175513 1158 webserver.cc:460] Webserver started at http://127.0.98.2:35821/ using document root <none> and password file <none>
I20260928 21:12:00.176096 1158 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:00.176187 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:00.176399 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:00.178507 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "3edce4f7eb534a2fafa49eba6f249b7c"
format_stamp: "Formatted at 2026-09-28 21:12:00 on dist-test-slave-v7xz"
server_key: "0d6c28ad98943997567a60f9a0f70468"
server_key_iv: "52cb2e89d89a1652b936abd62ab1b6d9"
server_key_version: "encryptionkey@0"
I20260928 21:12:00.179116 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "3edce4f7eb534a2fafa49eba6f249b7c"
format_stamp: "Formatted at 2026-09-28 21:12:00 on dist-test-slave-v7xz"
server_key: "0d6c28ad98943997567a60f9a0f70468"
server_key_iv: "52cb2e89d89a1652b936abd62ab1b6d9"
server_key_version: "encryptionkey@0"
I20260928 21:12:00.182914 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.002s
I20260928 21:12:00.185307 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.186614 1158 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:00.186765 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "3edce4f7eb534a2fafa49eba6f249b7c"
format_stamp: "Formatted at 2026-09-28 21:12:00 on dist-test-slave-v7xz"
server_key: "0d6c28ad98943997567a60f9a0f70468"
server_key_iv: "52cb2e89d89a1652b936abd62ab1b6d9"
server_key_version: "encryptionkey@0"
I20260928 21:12:00.186878 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:00.201146 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:00.204551 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:00.204748 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:00.205489 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:00.206310 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:00.206357 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.206434 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:00.206473 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.214581 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33241
I20260928 21:12:00.214648 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33241 every 8 connection(s)
I20260928 21:12:00.215440 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:00.222438 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1158
I20260928 21:12:00.222577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:00.222838 392 external_mini_cluster.cc:1428] Setting key 27460287b2be13bd7c504ad38add2e42
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:00.229887 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.217266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36995 (local address 127.0.98.62:43447)
0928 21:12:00.217402 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.217406 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.218264 (+ 858us) server_negotiation.cc:407] Connection header received
0928 21:12:00.219622 (+ 1358us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.219625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.219682 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.219785 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.220933 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.221460 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.222615 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.222882 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.223002 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.225789 (+ 2787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.225795 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.225803 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.225827 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.227345 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.228049 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.228053 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.228057 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.228158 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.228594 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.228597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.228598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.228778 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.228939 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.229655 (+ 716us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.229769 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:12:00.231164 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.217655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43447 (local address 127.0.98.2:36995)
0928 21:12:00.218115 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.218156 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.219404 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.219961 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.219968 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.220302 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:00.220759 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.220771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.221586 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.221589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.222490 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.222499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.223060 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.223065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.223274 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.223998 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:00.224015 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:00.225616 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:00.227471 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:00.227480 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:00.227500 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:00.227888 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:00.228270 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:00.228275 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:00.228279 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:00.228485 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:00.228947 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:00.228956 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:00.229344 (+ 388us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.229601 (+ 257us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.230012 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:00.232530 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43447
I20260928 21:12:00.232774 1287 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:00.233301 1287 heartbeater.cc:511] Master 127.0.98.62:43447 requested a full tablet report, sending...
I20260928 21:12:00.234513 962 ts_manager.cc:194] Registered new tserver with Master: 3edce4f7eb534a2fafa49eba6f249b7c (127.0.98.2:33241)
I20260928 21:12:00.235168 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36995
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:00.250041 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:00.360924 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:00.361176 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:00.361219 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:00.365453 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:00.365515 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:00.365593 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:00.370843 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43447
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:00.371809 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:00.372748 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:00.386085 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:00.386266 1292 server_base.cc:1034] running on GCE node
W20260928 21:12:00.386108 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:00.386108 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:00.386978 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:00.387686 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:00.388847 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790629920388792 us; error 55 us; skew 500 ppm
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:00.392639 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:00.393747 1292 webserver.cc:460] Webserver started at http://127.0.98.3:38339/ using document root <none> and password file <none>
I20260928 21:12:00.394353 1292 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:00.394441 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:00.394654 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:00.396688 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "4d59a12b80b44c4ea9eca9446a212672"
format_stamp: "Formatted at 2026-09-28 21:12:00 on dist-test-slave-v7xz"
server_key: "0a426c39ff5f4ed528b8bebceb4263b7"
server_key_iv: "521ded6fbcd372f07be68c5c317ea3b5"
server_key_version: "encryptionkey@0"
I20260928 21:12:00.397326 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4d59a12b80b44c4ea9eca9446a212672"
format_stamp: "Formatted at 2026-09-28 21:12:00 on dist-test-slave-v7xz"
server_key: "0a426c39ff5f4ed528b8bebceb4263b7"
server_key_iv: "521ded6fbcd372f07be68c5c317ea3b5"
server_key_version: "encryptionkey@0"
I20260928 21:12:00.401196 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:12:00.403702 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.405103 1292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:00.405227 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "4d59a12b80b44c4ea9eca9446a212672"
format_stamp: "Formatted at 2026-09-28 21:12:00 on dist-test-slave-v7xz"
server_key: "0a426c39ff5f4ed528b8bebceb4263b7"
server_key_iv: "521ded6fbcd372f07be68c5c317ea3b5"
server_key_version: "encryptionkey@0"
I20260928 21:12:00.405331 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:00.431334 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:00.434652 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:00.434854 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:00.435487 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:00.436291 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:00.436362 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.436432 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:00.436483 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:00.446966 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38739
I20260928 21:12:00.446985 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38739 every 8 connection(s)
I20260928 21:12:00.447839 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:00.456775 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1292
I20260928 21:12:00.456926 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:00.457211 392 external_mini_cluster.cc:1428] Setting key 20684613d57564ff02929496c168499d
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:00.464797 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.449704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41255 (local address 127.0.98.62:43447)
0928 21:12:00.449881 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.449889 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.450796 (+ 907us) server_negotiation.cc:407] Connection header received
0928 21:12:00.453018 (+ 2222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.453022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.453067 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.453145 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.455052 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.455579 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.457821 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.458127 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.458258 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.461247 (+ 2989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.461251 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.461254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.461279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.462783 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.463286 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.463289 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.463291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.463372 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.463728 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.463731 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.463732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.463917 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.464022 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.464504 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.464629 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:12:00.465639 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.450063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43447 (local address 127.0.98.3:41255)
0928 21:12:00.450586 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.450653 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.452721 (+ 2068us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.453394 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.453407 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.454031 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:00.454858 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.454883 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.455714 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.455720 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.457625 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.457642 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.458318 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.458324 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.458570 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.459285 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:00.459312 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:00.461092 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:00.462898 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:00.462903 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:00.462918 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:00.463178 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:00.463465 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:00.463470 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:00.463472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:00.463625 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:00.464018 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:00.464033 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:00.464273 (+ 240us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.464452 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.464771 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":105,"threads_started":1}
I20260928 21:12:00.466796 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43447
I20260928 21:12:00.467088 1421 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:00.467701 1421 heartbeater.cc:511] Master 127.0.98.62:43447 requested a full tablet report, sending...
I20260928 21:12:00.469063 962 ts_manager.cc:194] Registered new tserver with Master: 4d59a12b80b44c4ea9eca9446a212672 (127.0.98.3:38739)
I20260928 21:12:00.469700 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41255
I20260928 21:12:00.471568 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:12:00.483987 1001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.475372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40522 (local address 127.0.98.62:43447)
0928 21:12:00.475501 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.475506 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.475764 (+ 258us) server_negotiation.cc:407] Connection header received
0928 21:12:00.475931 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.475934 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.476013 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.476112 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.476908 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.477491 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.478419 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.478678 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.478801 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.480339 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.480343 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.480346 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.480384 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.482098 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.482586 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.482590 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.482592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.482687 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.482995 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.483000 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.483002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.483247 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.483402 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.483677 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.483804 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:12:00.489374 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:00.493263 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:00.510377 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.503181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39957 (local address 127.0.0.1:35236)
0928 21:12:00.504258 (+ 1077us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.504315 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.504459 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.505349 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.505357 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.505382 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:00.505570 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.505577 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.507091 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.507094 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.508479 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.508490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.509815 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.509818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.509925 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.509968 (+ 43us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.510089 (+ 121us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.510215 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":910,"mutex_wait_us":53,"thread_start_us":92,"threads_started":1}
I20260928 21:12:00.511046 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.503338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35236 (local address 127.0.98.1:39957)
0928 21:12:00.503758 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.503767 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.504617 (+ 850us) server_negotiation.cc:407] Connection header received
0928 21:12:00.504710 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.504718 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.504985 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.505160 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:00.505710 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.506931 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.508623 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.509484 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.509591 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.509754 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.510768 (+ 1014us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.510872 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":84,"threads_started":1}
I20260928 21:12:00.512126 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.503583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38739 (local address 127.0.0.1:46782)
0928 21:12:00.504404 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.504416 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.504480 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.506547 (+ 2067us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.506553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.506577 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:00.506719 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.506727 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.508532 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.508539 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.510057 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.510072 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.511492 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.511497 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.511638 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.511730 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.511799 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.511892 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":762,"mutex_wait_us":23,"thread_start_us":97,"threads_started":1}
I20260928 21:12:00.512780 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.503981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33241 (local address 127.0.0.1:37602)
0928 21:12:00.504678 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.504690 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.504754 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.505516 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.505520 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.505535 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:00.505634 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.505643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.507491 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.507507 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.509672 (+ 2165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.509682 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.511559 (+ 1877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.511563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.512380 (+ 817us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.512418 (+ 38us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.512488 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.512588 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"mutex_wait_us":685,"thread_start_us":94,"threads_started":1}
I20260928 21:12:00.513222 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.504625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.0.98.2:33241)
0928 21:12:00.505026 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.505038 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.505064 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:12:00.505142 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.505147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.505341 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.505857 (+ 516us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:00.506187 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.507346 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.509879 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.510981 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.511730 (+ 749us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.511879 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.512670 (+ 791us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.512773 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":121,"threads_started":1}
I20260928 21:12:00.513541 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.504203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46782 (local address 127.0.98.3:38739)
0928 21:12:00.506027 (+ 1824us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.506042 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.506069 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:12:00.506153 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.506161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.506370 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.506875 (+ 505us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:00.507068 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.508365 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.510333 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.511336 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.512988 (+ 1652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.513172 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.513229 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.513348 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1556,"thread_start_us":133,"threads_started":1}
I20260928 21:12:00.514521 1087 tablet_service.cc:1467] Processing CreateTablet for tablet aef5c59f461943f5b2f5d7fab1a922c9 (DEFAULT_TABLE table=test-table [id=e59190e0ca214a579a322479dc332f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:00.515666 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef5c59f461943f5b2f5d7fab1a922c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:00.516119 1221 tablet_service.cc:1467] Processing CreateTablet for tablet aef5c59f461943f5b2f5d7fab1a922c9 (DEFAULT_TABLE table=test-table [id=e59190e0ca214a579a322479dc332f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:00.516453 1355 tablet_service.cc:1467] Processing CreateTablet for tablet aef5c59f461943f5b2f5d7fab1a922c9 (DEFAULT_TABLE table=test-table [id=e59190e0ca214a579a322479dc332f51]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:00.517171 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef5c59f461943f5b2f5d7fab1a922c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:00.517539 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aef5c59f461943f5b2f5d7fab1a922c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:00.524660 1438 tablet_bootstrap.cc:492] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: Bootstrap starting.
I20260928 21:12:00.527120 1438 tablet_bootstrap.cc:654] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:00.527158 1440 tablet_bootstrap.cc:492] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: Bootstrap starting.
I20260928 21:12:00.527947 1438 log.cc:824] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:00.529770 1440 tablet_bootstrap.cc:654] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:00.529804 1438 tablet_bootstrap.cc:492] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: No bootstrap required, opened a new log
I20260928 21:12:00.530035 1438 ts_tablet_manager.cc:1397] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:12:00.531620 1440 log.cc:824] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:00.532436 1439 tablet_bootstrap.cc:492] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: Bootstrap starting.
I20260928 21:12:00.533658 1438 raft_consensus.cc:348] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.533913 1438 raft_consensus.cc:374] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:00.533967 1438 raft_consensus.cc:729] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f71868794dd4dbf83c6f81b38598204, State: Initialized, Role: FOLLOWER
I20260928 21:12:00.534499 1438 consensus_queue.cc:260] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.535090 1439 tablet_bootstrap.cc:654] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:00.535507 1438 ts_tablet_manager.cc:1428] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:12:00.535861 1153 heartbeater.cc:503] Master 127.0.98.62:43447 was elected leader, sending a full tablet report...
I20260928 21:12:00.536136 1440 tablet_bootstrap.cc:492] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: No bootstrap required, opened a new log
I20260928 21:12:00.536373 1440 ts_tablet_manager.cc:1397] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260928 21:12:00.536500 1439 log.cc:824] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:00.538690 1439 tablet_bootstrap.cc:492] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: No bootstrap required, opened a new log
I20260928 21:12:00.538894 1439 ts_tablet_manager.cc:1397] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:12:00.541508 1440 raft_consensus.cc:348] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.541781 1440 raft_consensus.cc:374] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:00.541939 1440 raft_consensus.cc:729] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d59a12b80b44c4ea9eca9446a212672, State: Initialized, Role: FOLLOWER
I20260928 21:12:00.542411 1440 consensus_queue.cc:260] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.543046 1421 heartbeater.cc:503] Master 127.0.98.62:43447 was elected leader, sending a full tablet report...
I20260928 21:12:00.543434 1440 ts_tablet_manager.cc:1428] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:12:00.544250 1439 raft_consensus.cc:348] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.544584 1439 raft_consensus.cc:374] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:00.544649 1439 raft_consensus.cc:729] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3edce4f7eb534a2fafa49eba6f249b7c, State: Initialized, Role: FOLLOWER
I20260928 21:12:00.545208 1439 consensus_queue.cc:260] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.546350 1439 ts_tablet_manager.cc:1428] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 21:12:00.546703 1287 heartbeater.cc:503] Master 127.0.98.62:43447 was elected leader, sending a full tablet report...
I20260928 21:12:00.682026 1445 raft_consensus.cc:482] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:00.682229 1445 raft_consensus.cc:504] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.683333 1445 leader_election.cc:288] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f71868794dd4dbf83c6f81b38598204 (127.0.98.1:39957), 3edce4f7eb534a2fafa49eba6f249b7c (127.0.98.2:33241)
I20260928 21:12:00.687908 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.683721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48683 (local address 127.0.98.1:39957)
0928 21:12:00.683849 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.683854 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.683907 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:12:00.684100 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.684104 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.684166 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.684258 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:00.684520 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.685270 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.686705 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.687335 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.687457 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.687528 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.687715 (+ 187us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.687799 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:12:00.687939 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.683648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39957 (local address 127.0.98.3:48683)
0928 21:12:00.683828 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.683845 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.683969 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.684289 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.684292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.684309 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:00.684407 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.684412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.685376 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.685379 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.686354 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.686369 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.687454 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.687457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.687586 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.687606 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.687671 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.687792 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":37}
I20260928 21:12:00.688635 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5c59f461943f5b2f5d7fab1a922c9" candidate_uuid: "4d59a12b80b44c4ea9eca9446a212672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f71868794dd4dbf83c6f81b38598204" is_pre_election: true
I20260928 21:12:00.688867 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.683964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33241 (local address 127.0.98.3:52165)
0928 21:12:00.684417 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:00.684428 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:00.684540 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:00.684873 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:00.684876 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:00.684893 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:00.685008 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.685013 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.685933 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.685936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.687598 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:00.687617 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.688508 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.688511 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:00.688579 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.688597 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:12:00.688662 (+ 65us) client_negotiation.cc:240] Negotiation successful
0928 21:12:00.688711 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"mutex_wait_us":3,"thread_start_us":163,"threads_started":1}
I20260928 21:12:00.688925 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.684049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52165 (local address 127.0.98.2:33241)
0928 21:12:00.684194 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.684199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.684533 (+ 334us) server_negotiation.cc:407] Connection header received
0928 21:12:00.684637 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.684639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.684730 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.684811 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:00.685119 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.685799 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.687758 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.688368 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.688475 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.688532 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.688714 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.688793 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:12:00.689056 1107 raft_consensus.cc:2454] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d59a12b80b44c4ea9eca9446a212672 in term 0.
I20260928 21:12:00.689591 1311 leader_election.cc:302] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d59a12b80b44c4ea9eca9446a212672, 8f71868794dd4dbf83c6f81b38598204; no voters:
I20260928 21:12:00.689625 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5c59f461943f5b2f5d7fab1a922c9" candidate_uuid: "4d59a12b80b44c4ea9eca9446a212672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" is_pre_election: true
I20260928 21:12:00.689874 1445 raft_consensus.cc:2781] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:00.689931 1241 raft_consensus.cc:2454] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d59a12b80b44c4ea9eca9446a212672 in term 0.
I20260928 21:12:00.689976 1445 raft_consensus.cc:482] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:00.690023 1445 raft_consensus.cc:3037] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:00.691270 1445 raft_consensus.cc:504] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.691733 1445 leader_election.cc:288] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [CANDIDATE]: Term 1 election: Requested vote from peers 8f71868794dd4dbf83c6f81b38598204 (127.0.98.1:39957), 3edce4f7eb534a2fafa49eba6f249b7c (127.0.98.2:33241)
I20260928 21:12:00.692082 1241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5c59f461943f5b2f5d7fab1a922c9" candidate_uuid: "4d59a12b80b44c4ea9eca9446a212672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3edce4f7eb534a2fafa49eba6f249b7c"
I20260928 21:12:00.692090 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aef5c59f461943f5b2f5d7fab1a922c9" candidate_uuid: "4d59a12b80b44c4ea9eca9446a212672" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f71868794dd4dbf83c6f81b38598204"
I20260928 21:12:00.692188 1241 raft_consensus.cc:3037] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:00.692234 1107 raft_consensus.cc:3037] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:00.693447 1107 raft_consensus.cc:2454] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d59a12b80b44c4ea9eca9446a212672 in term 1.
I20260928 21:12:00.693454 1241 raft_consensus.cc:2454] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d59a12b80b44c4ea9eca9446a212672 in term 1.
I20260928 21:12:00.693765 1311 leader_election.cc:302] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d59a12b80b44c4ea9eca9446a212672, 8f71868794dd4dbf83c6f81b38598204; no voters:
I20260928 21:12:00.693931 1445 raft_consensus.cc:2781] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:00.694147 1445 raft_consensus.cc:686] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [term 1 LEADER]: Becoming Leader. State: Replica: 4d59a12b80b44c4ea9eca9446a212672, State: Running, Role: LEADER
I20260928 21:12:00.694509 1445 consensus_queue.cc:237] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } }
I20260928 21:12:00.697772 961 catalog_manager.cc:5495] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4d59a12b80b44c4ea9eca9446a212672 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4d59a12b80b44c4ea9eca9446a212672" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f71868794dd4dbf83c6f81b38598204" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39957 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3edce4f7eb534a2fafa49eba6f249b7c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33241 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4d59a12b80b44c4ea9eca9446a212672" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38739 } health_report { overall_health: HEALTHY } } }
W20260928 21:12:00.698799 1422 tablet.cc:2369] T aef5c59f461943f5b2f5d7fab1a922c9 P 4d59a12b80b44c4ea9eca9446a212672: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:00.716599 1288 tablet.cc:2369] T aef5c59f461943f5b2f5d7fab1a922c9 P 3edce4f7eb534a2fafa49eba6f249b7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:00.754148 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.744889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35240 (local address 127.0.98.1:39957)
0928 21:12:00.745052 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.745057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.745266 (+ 209us) server_negotiation.cc:407] Connection header received
0928 21:12:00.745436 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.745439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.745502 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.745592 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.746280 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.746796 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.747639 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.747870 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.747985 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.750574 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.750587 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.750602 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.750629 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.752244 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.752968 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.752974 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.752976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.753058 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.753355 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.753358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.753359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.753607 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.753776 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.753919 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.754018 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:12:00.757004 1154 tablet.cc:2369] T aef5c59f461943f5b2f5d7fab1a922c9 P 8f71868794dd4dbf83c6f81b38598204: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:00.781831 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.772779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35246 (local address 127.0.98.1:39957)
0928 21:12:00.772982 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.772988 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.773189 (+ 201us) server_negotiation.cc:407] Connection header received
0928 21:12:00.773384 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.773387 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.773443 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.773533 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.774250 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.774806 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.775665 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.775910 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.776020 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.778600 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.778609 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.778611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.778637 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.780041 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.780483 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.780488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.780490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.780572 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.780876 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.780882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.780886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.781079 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.781216 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.781521 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.781648 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:12:00 dist-test-slave-v7xz krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:00.807658 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:00.799435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35250 (local address 127.0.98.1:39957)
0928 21:12:00.799602 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:00.799606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:00.799807 (+ 201us) server_negotiation.cc:407] Connection header received
0928 21:12:00.799988 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:00.799991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:00.800035 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:00.800127 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:00.800807 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.801371 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.802221 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:00.802438 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:00.802550 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:00.804905 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:00.804908 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:00.804910 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:00.804932 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:00.806119 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.806609 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.806613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.806614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.806703 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:00.806961 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:00.806963 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:00.806964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:00.807113 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:00.807215 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:00.807441 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:12:00.807537 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260928 21:12:00.809115 1067 tablet_service.cc:2908] Rejecting scan request for tablet aef5c59f461943f5b2f5d7fab1a922c9: Uninitialized: clean time has not yet been initialized
I20260928 21:12:00.811578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1024
I20260928 21:12:00.817484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1158
I20260928 21:12:00.822918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1292
I20260928 21:12:00.828266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 930
2026-09-28T21:12:00Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2464 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:01 dist-test-slave-v7xz krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:01 dist-test-slave-v7xz krb5kdc[1462](info): set up 2 sockets
Sep 28 21:12:01 dist-test-slave-v7xz krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260928 21:12:02.232609 392 mini_kdc.cc:121] Time spent starting KDC: real 1.379s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:02 dist-test-slave-v7xz krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:02.353924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:02.464470 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:02.464732 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:02.464773 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:02.468926 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:02.468981 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:12:02.468999 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:02.469018 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:02.469075 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:02.469110 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:02.474440 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:02.475454 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:02.476457 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:02.484867 1475 server_base.cc:1034] running on GCE node
W20260928 21:12:02.484890 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:02.485055 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:02.484889 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:02.485749 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:02.486773 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:02.487954 1475 hybrid_clock.cc:648] HybridClock initialized: now 1790629922487938 us; error 26 us; skew 500 ppm
Sep 28 21:12:02 dist-test-slave-v7xz krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:02.490703 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:02.491499 1475 webserver.cc:460] Webserver started at http://127.0.98.62:43903/ using document root <none> and password file <none>
I20260928 21:12:02.492004 1475 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:02.492063 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:02.492218 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:02.494264 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "60ee874dc4554325a28ce85bfabf99d3"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "29c3c3ad2c206fcb1cc7f55b374bd197"
server_key_iv: "70ee9dcd15709fb0d63d0ea7725a3440"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.494779 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "60ee874dc4554325a28ce85bfabf99d3"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "29c3c3ad2c206fcb1cc7f55b374bd197"
server_key_iv: "70ee9dcd15709fb0d63d0ea7725a3440"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.498323 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260928 21:12:02.500602 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:02.501896 1475 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:12:02.501992 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "60ee874dc4554325a28ce85bfabf99d3"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "29c3c3ad2c206fcb1cc7f55b374bd197"
server_key_iv: "70ee9dcd15709fb0d63d0ea7725a3440"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.502074 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:02.517678 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:02.520426 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:02.520589 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:02.528143 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45549
I20260928 21:12:02.528257 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45549 every 8 connection(s)
I20260928 21:12:02.529107 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:02.529815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1475
I20260928 21:12:02.529901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:02.530166 392 external_mini_cluster.cc:1428] Setting key 03e9e987060a45e136eddf711d61fbbd
I20260928 21:12:02.533082 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 28 21:12:02 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:02.539872 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: Bootstrap starting.
I20260928 21:12:02.542305 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:02.543378 1553 log.cc:824] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:02.544551 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:02.531459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37588 (local address 127.0.98.62:45549)
0928 21:12:02.531942 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:02.531957 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:02.531995 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:12:02.532639 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:02.532655 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:02.533078 (+ 423us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:02.533418 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:02.534271 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.535244 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.536393 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.536905 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.537025 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:02.539730 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:02.539740 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:02.539752 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:02.539790 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:02.541530 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:02.541987 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:02.541997 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:02.542002 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:02.542150 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:02.542439 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:02.542445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:02.542450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:02.542842 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:02.543051 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:02.543346 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:12:02.543662 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":141,"threads_started":1}
I20260928 21:12:02.545439 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: No bootstrap required, opened a new log
I20260928 21:12:02.548532 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ee874dc4554325a28ce85bfabf99d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260928 21:12:02.548774 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:02.548879 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60ee874dc4554325a28ce85bfabf99d3, State: Initialized, Role: FOLLOWER
I20260928 21:12:02.549273 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ee874dc4554325a28ce85bfabf99d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260928 21:12:02.549387 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:02.549432 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:02.549501 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:02.550307 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ee874dc4554325a28ce85bfabf99d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260928 21:12:02.550634 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60ee874dc4554325a28ce85bfabf99d3; no voters:
I20260928 21:12:02.550935 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:02.551061 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:02.551282 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [term 1 LEADER]: Becoming Leader. State: Replica: 60ee874dc4554325a28ce85bfabf99d3, State: Running, Role: LEADER
I20260928 21:12:02.551577 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ee874dc4554325a28ce85bfabf99d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } }
I20260928 21:12:02.552064 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:02.553153 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60ee874dc4554325a28ce85bfabf99d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ee874dc4554325a28ce85bfabf99d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260928 21:12:02.553278 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:02.553687 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:02.553951 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60ee874dc4554325a28ce85bfabf99d3. Latest consensus state: current_term: 1 leader_uuid: "60ee874dc4554325a28ce85bfabf99d3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60ee874dc4554325a28ce85bfabf99d3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45549 } } }
I20260928 21:12:02.554054 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:02.556322 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:02.561573 1567 catalog_manager.cc:1347] Generated new cluster ID: cf03c1223ea441fa9369ef6c425e71ae
I20260928 21:12:02.561640 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:02.584342 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:02.585765 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:02.597674 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: Generated new TSK 0
I20260928 21:12:02.598428 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:02.627214 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:02.738907 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:02.739174 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:02.739223 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:02.743269 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:02.743330 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:02.743412 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:02.743500 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:02.751487 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:02.752668 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:02.753877 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:02.763434 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:02.763453 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:02.763918 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:02.764210 1579 server_base.cc:1034] running on GCE node
I20260928 21:12:02.764604 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:02.765304 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:02.766496 1579 hybrid_clock.cc:648] HybridClock initialized: now 1790629922766472 us; error 49 us; skew 500 ppm
Sep 28 21:12:02 dist-test-slave-v7xz krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:02.769418 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:02.770367 1579 webserver.cc:460] Webserver started at http://127.0.98.1:45565/ using document root <none> and password file <none>
I20260928 21:12:02.770931 1579 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:02.770990 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:02.771204 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:02.773718 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "3747b3c6a5a049a59cd068f96b2c7068"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "0c6389edf040b88360e0a0de5658edb0"
server_key_iv: "f207a2ac7f2f9305d661022843b16328"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.774303 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3747b3c6a5a049a59cd068f96b2c7068"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "0c6389edf040b88360e0a0de5658edb0"
server_key_iv: "f207a2ac7f2f9305d661022843b16328"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.778260 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:02.780895 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:02.782238 1579 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:12:02.782379 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "3747b3c6a5a049a59cd068f96b2c7068"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "0c6389edf040b88360e0a0de5658edb0"
server_key_iv: "f207a2ac7f2f9305d661022843b16328"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.782497 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:02.797047 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:02.801038 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:02.801230 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:02.802938 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:02.803004 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:02.803077 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:02.803110 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:02.814332 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34585
I20260928 21:12:02.814399 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34585 every 8 connection(s)
I20260928 21:12:02.815366 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:12:02 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:02.820434 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:02.803939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43197 (local address 127.0.98.62:45549)
0928 21:12:02.804055 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:02.804061 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:02.805258 (+ 1197us) server_negotiation.cc:407] Connection header received
0928 21:12:02.806760 (+ 1502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:02.806767 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:02.806840 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:02.806905 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:02.808918 (+ 2013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.809830 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.811734 (+ 1904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.812133 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.812230 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:02.817275 (+ 5045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:02.817279 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:02.817282 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:02.817309 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:02.818694 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:02.819156 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:02.819159 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:02.819160 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:02.819229 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:02.819514 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:02.819517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:02.819518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:02.819719 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:02.819793 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:02.820218 (+ 425us) server_negotiation.cc:299] Negotiation successful
0928 21:12:02.820311 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:12:02.821319 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:02.804510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:43197)
0928 21:12:02.805095 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:02.805158 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:02.806477 (+ 1319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:02.807361 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:02.807371 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:02.807870 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:02.808701 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:02.808727 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.810007 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.810010 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:02.811565 (+ 1555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:02.811578 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.812454 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.812461 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:02.812710 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:02.815070 (+ 2360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:02.815088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:02.817105 (+ 2017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:02.818798 (+ 1693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:02.818803 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:02.818813 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:02.819059 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:02.819303 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:02.819305 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:02.819306 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:02.819431 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:02.819803 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:02.819807 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:02.820022 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 21:12:02.820162 (+ 140us) client_negotiation.cc:240] Negotiation successful
0928 21:12:02.820408 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":140,"threads_started":1}
I20260928 21:12:02.822583 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260928 21:12:02.822751 1710 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:02.823228 1710 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260928 21:12:02.824785 1505 ts_manager.cc:194] Registered new tserver with Master: 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585)
I20260928 21:12:02.825186 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1579
I20260928 21:12:02.825317 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:02.825657 392 external_mini_cluster.cc:1428] Setting key 2649a3c7da6a92a94aca8af47c72c79a
I20260928 21:12:02.826261 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43197
I20260928 21:12:02.832948 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:02.824665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.1:60893)
0928 21:12:02.824975 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:02.824991 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:02.825085 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:02.825430 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:02.825433 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:02.825652 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:02.825787 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:02.825795 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.826841 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.826844 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:02.827767 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:02.827782 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.828349 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.828355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:02.828480 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:02.829065 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:02.829085 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:02.829567 (+ 482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:02.831458 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:02.831461 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:02.831462 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:02.831649 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:02.831945 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:02.831947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:02.831948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:02.832020 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:02.832475 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:02.832477 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:02.832548 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 21:12:02.832623 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:12:02.832736 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260928 21:12:02.833132 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:02.824731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60893 (local address 127.0.98.62:45549)
0928 21:12:02.824872 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:02.824876 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:02.825069 (+ 193us) server_negotiation.cc:407] Connection header received
0928 21:12:02.825170 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:02.825174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:02.825225 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:02.825312 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:02.825908 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.826600 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.827907 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:02.828219 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:02.828348 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:02.829697 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:02.829703 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:02.829709 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:02.829739 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:02.831349 (+ 1610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:02.831747 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:02.831753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:02.831758 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:02.831848 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:02.832112 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:02.832118 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:02.832122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:02.832370 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:02.832481 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:02.832717 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 21:12:02.832898 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:02.852651 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:02.960314 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:02.960574 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:02.960620 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:02.964658 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:02.964716 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:02.964789 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:02.964869 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:02.970144 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:02.971174 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:02.972158 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:02.981333 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:02.981396 1719 server_base.cc:1034] running on GCE node
W20260928 21:12:02.981340 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:02.981340 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:02.981940 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:02.982553 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:02.983757 1719 hybrid_clock.cc:648] HybridClock initialized: now 1790629922983744 us; error 32 us; skew 500 ppm
Sep 28 21:12:02 dist-test-slave-v7xz krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:02.986701 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:02.987586 1719 webserver.cc:460] Webserver started at http://127.0.98.2:41623/ using document root <none> and password file <none>
I20260928 21:12:02.988094 1719 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:02.988152 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:02.988309 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:02.990290 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "e24fa22fc3704ac1856006a5c08cda7e"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "349a2b810c9612f6258d15a64b86655a"
server_key_iv: "b549e2158d7442ca2593a1a20270c4d0"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.990779 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e24fa22fc3704ac1856006a5c08cda7e"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "349a2b810c9612f6258d15a64b86655a"
server_key_iv: "b549e2158d7442ca2593a1a20270c4d0"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.994382 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:12:02.996673 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:02.997951 1719 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:02.998097 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "e24fa22fc3704ac1856006a5c08cda7e"
format_stamp: "Formatted at 2026-09-28 21:12:02 on dist-test-slave-v7xz"
server_key: "349a2b810c9612f6258d15a64b86655a"
server_key_iv: "b549e2158d7442ca2593a1a20270c4d0"
server_key_version: "encryptionkey@0"
I20260928 21:12:02.998198 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:03.010294 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:03.013338 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:03.013496 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:03.014823 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:03.014873 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:03.014935 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:03.014968 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:03 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:03.025832 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40245
I20260928 21:12:03.025902 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40245 every 8 connection(s)
I20260928 21:12:03.026722 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:03.028489 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1719
I20260928 21:12:03.028604 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:03.028973 392 external_mini_cluster.cc:1428] Setting key 1eb001ab26bc38dc0fa73f8c61ac4f70
I20260928 21:12:03.029703 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.015657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56733 (local address 127.0.98.62:45549)
0928 21:12:03.015881 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.015885 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.016856 (+ 971us) server_negotiation.cc:407] Connection header received
0928 21:12:03.018007 (+ 1151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.018012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.018087 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.018190 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:03.019880 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.020776 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.022304 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.022696 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.022814 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.025698 (+ 2884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:03.025702 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:03.025705 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:03.025738 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:03.027416 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.027936 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.027938 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.027939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.028020 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.028339 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.028342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.028347 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.028601 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:03.028704 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.029415 (+ 711us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.029565 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260928 21:12:03.030586 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.016200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:56733)
0928 21:12:03.016677 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.016731 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.017725 (+ 994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.018507 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.018517 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.018953 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:03.019695 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.019715 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.020959 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.020965 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.022175 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.022184 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.022979 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.022982 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.023126 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.023744 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:03.023783 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:03.025465 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:03.027530 (+ 2065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.027535 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.027548 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.027847 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.028115 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.028118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.028119 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.028247 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.028764 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:03.028770 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:03.029104 (+ 334us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.029318 (+ 214us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.029651 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":144,"threads_started":1}
I20260928 21:12:03.031927 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260928 21:12:03.032060 1850 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:03.032692 1850 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260928 21:12:03.033844 1507 ts_manager.cc:194] Registered new tserver with Master: e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2:40245)
I20260928 21:12:03.034713 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56733
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:03.042810 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.034277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.2:33627)
0928 21:12:03.034601 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.034613 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.034728 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.035046 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.035051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.035304 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:03.035461 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.035471 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.036480 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.036486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.037508 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.037520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.037953 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.037957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.038067 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.038641 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:03.038655 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:03.039083 (+ 428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:03.040954 (+ 1871us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.040957 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.040958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.041228 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.041537 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.041543 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.041545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.041639 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.042230 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:03.042233 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:03.042336 (+ 103us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.042451 (+ 115us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.042624 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":106,"threads_started":1}
I20260928 21:12:03.042801 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.034318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33627 (local address 127.0.98.62:45549)
0928 21:12:03.034438 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.034442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.034688 (+ 246us) server_negotiation.cc:407] Connection header received
0928 21:12:03.034875 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.034878 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.034926 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.035028 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:03.035596 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.036336 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.037633 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.037837 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.037917 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.039219 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:03.039222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:03.039224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:03.039245 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:03.040783 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.041318 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.041320 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.041321 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.041395 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.041814 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.041820 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.041825 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.042118 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:03.042238 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.042508 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.042629 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:03.057770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45549
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33069
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:03.169580 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:03.169915 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:03.169996 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:03.174340 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:03.174423 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:03.174492 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:03.174580 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:03.180195 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33069
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45549
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:03.181380 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:03.182421 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:03.192265 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:03.192201 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:03.192298 1859 server_base.cc:1034] running on GCE node
W20260928 21:12:03.192201 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:03.192981 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:03.193629 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:03.194820 1859 hybrid_clock.cc:648] HybridClock initialized: now 1790629923194810 us; error 46 us; skew 500 ppm
Sep 28 21:12:03 dist-test-slave-v7xz krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:03.197562 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:03.198431 1859 webserver.cc:460] Webserver started at http://127.0.98.3:38567/ using document root <none> and password file <none>
I20260928 21:12:03.198954 1859 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:03.199011 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:03.199169 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:03.201179 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "d239c7a25bd14cb885d4ef1c98682d62"
format_stamp: "Formatted at 2026-09-28 21:12:03 on dist-test-slave-v7xz"
server_key: "533532b6044e60aa9ad07114c9553fd8"
server_key_iv: "253ad87fce5f104b36e124e8a81195de"
server_key_version: "encryptionkey@0"
I20260928 21:12:03.201678 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d239c7a25bd14cb885d4ef1c98682d62"
format_stamp: "Formatted at 2026-09-28 21:12:03 on dist-test-slave-v7xz"
server_key: "533532b6044e60aa9ad07114c9553fd8"
server_key_iv: "253ad87fce5f104b36e124e8a81195de"
server_key_version: "encryptionkey@0"
I20260928 21:12:03.205552 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:12:03.208006 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:03.209293 1859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:03.209384 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "d239c7a25bd14cb885d4ef1c98682d62"
format_stamp: "Formatted at 2026-09-28 21:12:03 on dist-test-slave-v7xz"
server_key: "533532b6044e60aa9ad07114c9553fd8"
server_key_iv: "253ad87fce5f104b36e124e8a81195de"
server_key_version: "encryptionkey@0"
I20260928 21:12:03.209510 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:03.225982 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:03.228778 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:03.229038 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:03.230468 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:03.230515 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:03.230594 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:03.230641 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:03 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:03.241417 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39181
I20260928 21:12:03.241523 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39181 every 8 connection(s)
I20260928 21:12:03.242348 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:03.244077 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 1859
I20260928 21:12:03.244120 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.231183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36503 (local address 127.0.98.62:45549)
0928 21:12:03.231340 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.231345 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.232309 (+ 964us) server_negotiation.cc:407] Connection header received
0928 21:12:03.233450 (+ 1141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.233455 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.233516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.233571 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:03.234851 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.235443 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.236642 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.236942 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.237063 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.240396 (+ 3333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:03.240399 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:03.240402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:03.240426 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:03.242029 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.242532 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.242536 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.242538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.242632 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.243023 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.243027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.243028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.243232 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:03.243330 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.243807 (+ 477us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.243947 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:12:03.244199 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:03.244462 392 external_mini_cluster.cc:1428] Setting key 791f189c2e644a80b0fa5b3ee37f15f2
I20260928 21:12:03.244939 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.231639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:36503)
0928 21:12:03.232154 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.232209 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.233204 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.233945 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.233952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.234262 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:03.234712 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.234724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.235584 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.235587 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.236511 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.236520 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.237098 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.237102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.237261 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.237868 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:03.237891 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:03.240233 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:03.242150 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.242157 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.242171 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.242422 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.242753 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.242756 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.242758 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.242910 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.243343 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:03.243349 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:03.243605 (+ 256us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.243760 (+ 155us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.243989 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":178,"threads_started":1}
I20260928 21:12:03.246342 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45549
I20260928 21:12:03.246572 1990 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:03.247126 1990 heartbeater.cc:511] Master 127.0.98.62:45549 requested a full tablet report, sending...
I20260928 21:12:03.248137 1505 ts_manager.cc:194] Registered new tserver with Master: d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:03.248735 1505 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:36503
I20260928 21:12:03.255465 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.248160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.98.3:52807)
0928 21:12:03.249094 (+ 934us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.249107 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.249215 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.249452 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.249455 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.249654 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:03.249760 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.249768 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.250497 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.250500 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.251091 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.251099 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.251469 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.251471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.251533 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.252050 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:03.252061 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:03.252409 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:03.254177 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.254180 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.254181 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.254431 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.254684 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.254691 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.254692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.254764 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.255078 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:03.255080 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:03.255152 (+ 72us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.255233 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.255328 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":835,"thread_start_us":94,"threads_started":1}
I20260928 21:12:03.255476 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.248265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52807 (local address 127.0.98.62:45549)
0928 21:12:03.248382 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.248386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.249178 (+ 792us) server_negotiation.cc:407] Connection header received
0928 21:12:03.249314 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.249316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.249354 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.249417 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:03.249898 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.250370 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.251188 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.251381 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.251453 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.252545 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:03.252548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:03.252550 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:03.252570 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:03.254077 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.254513 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.254515 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.254517 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.254584 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.254840 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.254842 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.254843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.254992 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:03.255066 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.255289 (+ 223us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.255372 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:12:03.258286 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:12:03 dist-test-slave-v7xz krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:12:03 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:03.282892 1556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.275148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37594 (local address 127.0.98.62:45549)
0928 21:12:03.275275 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.275279 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.275291 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:03.275329 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.275331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.275368 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.275433 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:03.276114 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.276607 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.277453 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.277656 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.277740 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.280114 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:03.280117 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:03.280119 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:03.280142 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:03.281484 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.281872 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.281874 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.281875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.281944 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.282226 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.282229 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.282230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.282392 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:03.282493 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.282676 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.282771 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:03.285768 1505 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:03.288818 1505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:03.303435 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.297115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40245 (local address 127.0.0.1:38526)
0928 21:12:03.298206 (+ 1091us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.298220 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.298311 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.298990 (+ 679us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.298994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.299009 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.299200 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.299210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.300728 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.300732 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.301984 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.301990 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.302868 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.302871 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.302983 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.303038 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.303168 (+ 130us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.303291 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1038,"mutex_wait_us":53,"spinlock_wait_cycles":8448,"thread_start_us":124,"threads_started":1}
I20260928 21:12:03.303443 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.297829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38526 (local address 127.0.98.2:40245)
0928 21:12:03.298369 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.298382 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.298401 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:12:03.298462 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.298466 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.298676 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.298835 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.299340 (+ 505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.300620 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.302103 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.302740 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.302858 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.303019 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.303200 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.303291 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":109,"threads_started":1}
I20260928 21:12:03.303651 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.297433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39181 (local address 127.0.0.1:58022)
0928 21:12:03.298324 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.298337 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.298413 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.299005 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.299008 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.299025 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.299200 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.299205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.300149 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.300156 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.301802 (+ 1646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.301812 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.303088 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.303092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.303301 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.303349 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.303421 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.303487 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":844,"mutex_wait_us":205,"thread_start_us":85,"threads_started":1}
I20260928 21:12:03.304258 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.297815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58022 (local address 127.0.98.3:39181)
0928 21:12:03.298206 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.298218 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.298501 (+ 283us) server_negotiation.cc:407] Connection header received
0928 21:12:03.298571 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.298575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.298723 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.298883 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.299309 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.300037 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.301937 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.302748 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.302859 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.303023 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.304008 (+ 985us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.304110 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":88,"threads_started":1}
I20260928 21:12:03.304270 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.297156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.0.98.1:34585)
0928 21:12:03.297897 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.297914 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.298374 (+ 460us) server_negotiation.cc:407] Connection header received
0928 21:12:03.298458 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.298463 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.298676 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.298835 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.299138 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.300358 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.301889 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.302762 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.302859 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.303023 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.304059 (+ 1036us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.304143 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":87,"threads_started":1}
I20260928 21:12:03.304548 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.297052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34585 (local address 127.0.0.1:35416)
0928 21:12:03.298028 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.298078 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.298214 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.298844 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.298850 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.298871 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.299004 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.299011 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.300506 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.300509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.301765 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.301774 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.303088 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.303092 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.303267 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.303349 (+ 82us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.303421 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.304460 (+ 1039us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"mutex_wait_us":82,"thread_start_us":63,"threads_started":1}
I20260928 21:12:03.306474 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 177d74f71fd34d33aa3c1c00bd917b5c (DEFAULT_TABLE table=test-table [id=eb7c3fd701724906ada79dfba449fda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:03.306846 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 177d74f71fd34d33aa3c1c00bd917b5c (DEFAULT_TABLE table=test-table [id=eb7c3fd701724906ada79dfba449fda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:03.307472 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177d74f71fd34d33aa3c1c00bd917b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.307821 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177d74f71fd34d33aa3c1c00bd917b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.313800 2013 tablet_bootstrap.cc:492] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Bootstrap starting.
I20260928 21:12:03.313941 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 177d74f71fd34d33aa3c1c00bd917b5c (DEFAULT_TABLE table=test-table [id=eb7c3fd701724906ada79dfba449fda9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:03.314955 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 177d74f71fd34d33aa3c1c00bd917b5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.315397 2014 tablet_bootstrap.cc:492] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Bootstrap starting.
I20260928 21:12:03.315929 2013 tablet_bootstrap.cc:654] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.316742 2013 log.cc:824] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:03.317548 2014 tablet_bootstrap.cc:654] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.318428 2014 log.cc:824] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:03.318707 2013 tablet_bootstrap.cc:492] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: No bootstrap required, opened a new log
I20260928 21:12:03.318917 2013 ts_tablet_manager.cc:1397] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:12:03.320401 2014 tablet_bootstrap.cc:492] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: No bootstrap required, opened a new log
I20260928 21:12:03.320588 2014 ts_tablet_manager.cc:1397] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260928 21:12:03.321998 2013 raft_consensus.cc:348] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.322341 2013 raft_consensus.cc:374] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.322456 2017 tablet_bootstrap.cc:492] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Bootstrap starting.
I20260928 21:12:03.322474 2013 raft_consensus.cc:729] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d239c7a25bd14cb885d4ef1c98682d62, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.322904 2013 consensus_queue.cc:260] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.323624 1990 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260928 21:12:03.324379 2013 ts_tablet_manager.cc:1428] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260928 21:12:03.324590 2014 raft_consensus.cc:348] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.324920 2014 raft_consensus.cc:374] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.325119 2014 raft_consensus.cc:729] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24fa22fc3704ac1856006a5c08cda7e, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.325528 2017 tablet_bootstrap.cc:654] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.325605 2014 consensus_queue.cc:260] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.326236 1850 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260928 21:12:03.326386 2017 log.cc:824] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:03.326574 2014 ts_tablet_manager.cc:1428] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260928 21:12:03.328373 2017 tablet_bootstrap.cc:492] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: No bootstrap required, opened a new log
I20260928 21:12:03.328568 2017 ts_tablet_manager.cc:1397] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:12:03.331517 2017 raft_consensus.cc:348] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.331740 2017 raft_consensus.cc:374] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.331799 2017 raft_consensus.cc:729] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3747b3c6a5a049a59cd068f96b2c7068, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.332212 2017 consensus_queue.cc:260] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.332916 1710 heartbeater.cc:503] Master 127.0.98.62:45549 was elected leader, sending a full tablet report...
I20260928 21:12:03.333209 2017 ts_tablet_manager.cc:1428] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260928 21:12:03.493429 1991 tablet.cc:2369] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:03.527962 1851 tablet.cc:2369] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:03.567047 1711 tablet.cc:2369] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:03.675025 2021 raft_consensus.cc:482] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:03.675276 2021 raft_consensus.cc:504] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.676522 2021 leader_election.cc:288] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2:40245), d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:03.680210 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.676866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59315 (local address 127.0.98.2:40245)
0928 21:12:03.676999 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.677004 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.677158 (+ 154us) server_negotiation.cc:407] Connection header received
0928 21:12:03.677323 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.677328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.677375 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.677460 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.677716 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.678209 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.679279 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.679667 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.679786 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.679846 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.680021 (+ 175us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.680088 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:12:03.680228 2023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.676714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40245 (local address 127.0.98.1:59315)
0928 21:12:03.677061 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.677073 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.677217 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.677473 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.677476 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.677491 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.677611 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.677616 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.678315 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.678318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.679156 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.679165 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.679797 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.679800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.679894 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.679921 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.680006 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.680082 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"mutex_wait_us":24,"thread_start_us":105,"threads_started":1}
I20260928 21:12:03.680333 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.676931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33761 (local address 127.0.98.3:39181)
0928 21:12:03.677056 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.677060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.677258 (+ 198us) server_negotiation.cc:407] Connection header received
0928 21:12:03.677411 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.677414 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.677462 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.677546 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.677836 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.678342 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.679378 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.679753 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.679873 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.679936 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.680120 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.680186 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:03.680387 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.676801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39181 (local address 127.0.98.1:33761)
0928 21:12:03.677160 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.677171 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.677299 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.677558 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.677561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.677576 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.677688 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.677694 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.678458 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.678462 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.679253 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.679261 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.679874 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.679878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.679994 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.680018 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.680106 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.680170 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":18,"thread_start_us":73,"threads_started":1}
I20260928 21:12:03.680820 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177d74f71fd34d33aa3c1c00bd917b5c" candidate_uuid: "3747b3c6a5a049a59cd068f96b2c7068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24fa22fc3704ac1856006a5c08cda7e" is_pre_election: true
I20260928 21:12:03.681137 1804 raft_consensus.cc:2454] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3747b3c6a5a049a59cd068f96b2c7068 in term 0.
I20260928 21:12:03.681596 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177d74f71fd34d33aa3c1c00bd917b5c" candidate_uuid: "3747b3c6a5a049a59cd068f96b2c7068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d239c7a25bd14cb885d4ef1c98682d62" is_pre_election: true
I20260928 21:12:03.681608 1595 leader_election.cc:302] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3747b3c6a5a049a59cd068f96b2c7068, e24fa22fc3704ac1856006a5c08cda7e; no voters:
I20260928 21:12:03.681864 1944 raft_consensus.cc:2454] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3747b3c6a5a049a59cd068f96b2c7068 in term 0.
I20260928 21:12:03.681891 2021 raft_consensus.cc:2781] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:03.681969 2021 raft_consensus.cc:482] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:03.682011 2021 raft_consensus.cc:3037] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:03.683141 2021 raft_consensus.cc:504] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.683452 2021 leader_election.cc:288] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [CANDIDATE]: Term 1 election: Requested vote from peers e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2:40245), d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:03.683768 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177d74f71fd34d33aa3c1c00bd917b5c" candidate_uuid: "3747b3c6a5a049a59cd068f96b2c7068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e24fa22fc3704ac1856006a5c08cda7e"
I20260928 21:12:03.683763 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "177d74f71fd34d33aa3c1c00bd917b5c" candidate_uuid: "3747b3c6a5a049a59cd068f96b2c7068" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d239c7a25bd14cb885d4ef1c98682d62"
I20260928 21:12:03.683880 1944 raft_consensus.cc:3037] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:03.683887 1804 raft_consensus.cc:3037] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:03.685276 1804 raft_consensus.cc:2454] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3747b3c6a5a049a59cd068f96b2c7068 in term 1.
I20260928 21:12:03.685608 1944 raft_consensus.cc:2454] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3747b3c6a5a049a59cd068f96b2c7068 in term 1.
I20260928 21:12:03.685611 1595 leader_election.cc:302] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3747b3c6a5a049a59cd068f96b2c7068, e24fa22fc3704ac1856006a5c08cda7e; no voters:
I20260928 21:12:03.685847 2021 raft_consensus.cc:2781] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:03.686066 2021 raft_consensus.cc:686] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 LEADER]: Becoming Leader. State: Replica: 3747b3c6a5a049a59cd068f96b2c7068, State: Running, Role: LEADER
I20260928 21:12:03.686378 2021 consensus_queue.cc:237] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.689356 1505 catalog_manager.cc:5495] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3747b3c6a5a049a59cd068f96b2c7068" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:03.757418 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.753698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38698 (local address 127.0.98.1:34585)
0928 21:12:03.753837 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.753841 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.753855 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:03.753992 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.753994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.754048 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.754132 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:03.754401 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.754912 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.755835 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.756067 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.756182 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.756520 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:03.756994 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:03.757114 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.757207 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.757264 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:12:03.763118 1804 raft_consensus.cc:1261] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Refusing update from remote peer 3747b3c6a5a049a59cd068f96b2c7068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:03.763132 1944 raft_consensus.cc:1261] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Refusing update from remote peer 3747b3c6a5a049a59cd068f96b2c7068: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:03.763769 2021 consensus_queue.cc:1035] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:03.763970 2024 consensus_queue.cc:1035] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:03.771531 2028 mvcc.cc:205] Tried to move back new op lower bound from 7334420167727009792 to 7334420167421849600. Current Snapshot: MvccSnapshot[applied={T|T < 7334420167727009792}]
I20260928 21:12:03.783034 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:03.783221 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:03.785806 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: Sending DeleteTablet for 3 replicas of tablet 177d74f71fd34d33aa3c1c00bd917b5c
I20260928 21:12:03.786615 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 177d74f71fd34d33aa3c1c00bd917b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38526
I20260928 21:12:03.786610 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 177d74f71fd34d33aa3c1c00bd917b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58022
I20260928 21:12:03.786646 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 177d74f71fd34d33aa3c1c00bd917b5c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:03 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35416
I20260928 21:12:03.787216 2039 tablet_replica.cc:330] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: stopping tablet replica
I20260928 21:12:03.787305 2040 tablet_replica.cc:330] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: stopping tablet replica
I20260928 21:12:03.787590 2039 raft_consensus.cc:2229] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:03.787629 2040 raft_consensus.cc:2229] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:03.787935 2039 raft_consensus.cc:2258] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:03.788061 2040 raft_consensus.cc:2258] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:03.788125 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:12:03.788296 2038 tablet_replica.cc:330] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: stopping tablet replica
W20260928 21:12:03.788545 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:03.788789 2038 raft_consensus.cc:2229] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:03.789093 2038 raft_consensus.cc:2258] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:03.790266 2040 ts_tablet_manager.cc:1905] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:03.790418 2038 ts_tablet_manager.cc:1905] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:03.793075 2038 ts_tablet_manager.cc:1918] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:03.793196 2038 log.cc:1196] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/177d74f71fd34d33aa3c1c00bd917b5c
I20260928 21:12:03.793265 1644 tablet_service.cc:1467] Processing CreateTablet for tablet dde12d022d454fa49464f1a7439e58cb (DEFAULT_TABLE table=test-table [id=e275d244a8824aaeb1fefa36ac9207c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:03.793522 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde12d022d454fa49464f1a7439e58cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.793526 2038 ts_tablet_manager.cc:1939] T 177d74f71fd34d33aa3c1c00bd917b5c P d239c7a25bd14cb885d4ef1c98682d62: Deleting consensus metadata
I20260928 21:12:03.793581 1924 tablet_service.cc:1467] Processing CreateTablet for tablet dde12d022d454fa49464f1a7439e58cb (DEFAULT_TABLE table=test-table [id=e275d244a8824aaeb1fefa36ac9207c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:03.793805 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde12d022d454fa49464f1a7439e58cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.795512 2039 ts_tablet_manager.cc:1905] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:03.796651 1493 catalog_manager.cc:4841] TS d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181): tablet 177d74f71fd34d33aa3c1c00bd917b5c (table test-table [id=eb7c3fd701724906ada79dfba449fda9]) successfully deleted
I20260928 21:12:03.796931 2017 tablet_bootstrap.cc:492] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Bootstrap starting.
I20260928 21:12:03.797711 2017 tablet_bootstrap.cc:654] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.798106 2039 ts_tablet_manager.cc:1918] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:03.798200 2039 log.cc:1196] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/177d74f71fd34d33aa3c1c00bd917b5c
I20260928 21:12:03.798548 2039 ts_tablet_manager.cc:1939] T 177d74f71fd34d33aa3c1c00bd917b5c P e24fa22fc3704ac1856006a5c08cda7e: Deleting consensus metadata
I20260928 21:12:03.798900 1784 tablet_service.cc:1467] Processing CreateTablet for tablet dde12d022d454fa49464f1a7439e58cb (DEFAULT_TABLE table=test-table [id=e275d244a8824aaeb1fefa36ac9207c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:03.799197 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dde12d022d454fa49464f1a7439e58cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.799820 1491 catalog_manager.cc:4841] TS e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2:40245): tablet 177d74f71fd34d33aa3c1c00bd917b5c (table test-table [id=eb7c3fd701724906ada79dfba449fda9]) successfully deleted
I20260928 21:12:03.800542 2017 tablet_bootstrap.cc:492] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: No bootstrap required, opened a new log
I20260928 21:12:03.800629 2017 ts_tablet_manager.cc:1397] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260928 21:12:03.801142 2014 tablet_bootstrap.cc:492] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Bootstrap starting.
I20260928 21:12:03.801096 2017 raft_consensus.cc:348] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.801203 2017 raft_consensus.cc:374] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.801247 2017 raft_consensus.cc:729] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3747b3c6a5a049a59cd068f96b2c7068, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.801379 2017 consensus_queue.cc:260] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.801640 2017 ts_tablet_manager.cc:1428] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20260928 21:12:03.801826 2013 tablet_bootstrap.cc:492] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Bootstrap starting.
I20260928 21:12:03.802099 2014 tablet_bootstrap.cc:654] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.802704 2013 tablet_bootstrap.cc:654] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.804556 2013 tablet_bootstrap.cc:492] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: No bootstrap required, opened a new log
I20260928 21:12:03.804634 2013 ts_tablet_manager.cc:1397] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260928 21:12:03.805089 2013 raft_consensus.cc:348] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.805189 2013 raft_consensus.cc:374] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.805219 2013 raft_consensus.cc:729] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d239c7a25bd14cb885d4ef1c98682d62, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.805222 2040 ts_tablet_manager.cc:1918] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:03.805317 2040 log.cc:1196] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/177d74f71fd34d33aa3c1c00bd917b5c
I20260928 21:12:03.805354 2013 consensus_queue.cc:260] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.805617 2013 ts_tablet_manager.cc:1428] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:12:03.805684 2040 ts_tablet_manager.cc:1939] T 177d74f71fd34d33aa3c1c00bd917b5c P 3747b3c6a5a049a59cd068f96b2c7068: Deleting consensus metadata
I20260928 21:12:03.805787 2014 tablet_bootstrap.cc:492] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: No bootstrap required, opened a new log
I20260928 21:12:03.805886 2014 ts_tablet_manager.cc:1397] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260928 21:12:03.806329 2014 raft_consensus.cc:348] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.806437 2014 raft_consensus.cc:374] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.806478 2014 raft_consensus.cc:729] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24fa22fc3704ac1856006a5c08cda7e, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.806599 2014 consensus_queue.cc:260] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.806834 2014 ts_tablet_manager.cc:1428] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:12:03.807016 1492 catalog_manager.cc:4841] TS 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585): tablet 177d74f71fd34d33aa3c1c00bd917b5c (table test-table [id=eb7c3fd701724906ada79dfba449fda9]) successfully deleted
I20260928 21:12:03.843048 2019 raft_consensus.cc:482] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:03.843220 2019 raft_consensus.cc:504] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.844347 2019 leader_election.cc:288] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585), d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:03.849210 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.844941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53031 (local address 127.0.98.1:34585)
0928 21:12:03.845057 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.845061 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.845076 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:03.845235 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.845238 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.845288 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.845374 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.845733 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.846597 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.847842 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.848447 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.848559 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.848627 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.848996 (+ 369us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.849089 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:12:03.849501 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.844745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39279 (local address 127.0.98.3:39181)
0928 21:12:03.844898 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.844903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.845126 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:12:03.845302 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.845304 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.845364 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.845436 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:03.845836 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.846588 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.848213 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.848846 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.848923 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.848982 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.849309 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.849391 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:12:03.849502 2045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.844584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39181 (local address 127.0.98.2:39279)
0928 21:12:03.845050 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.845063 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.845185 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.845521 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.845525 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.845544 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.845709 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.845716 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.846698 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.846702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.848075 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.848085 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.849070 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.849073 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.849173 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.849192 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.849271 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.849339 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"mutex_wait_us":59,"thread_start_us":106,"threads_started":1}
I20260928 21:12:03.849980 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.844576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34585 (local address 127.0.98.2:53031)
0928 21:12:03.844961 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.844977 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.845104 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.845401 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.845407 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.845430 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:03.845609 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.845616 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.846738 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.846748 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.847713 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.847723 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.848700 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.848703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.848848 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.848874 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.849430 (+ 556us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.849502 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":55,"thread_start_us":72,"threads_started":1}
I20260928 21:12:03.850152 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde12d022d454fa49464f1a7439e58cb" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d239c7a25bd14cb885d4ef1c98682d62" is_pre_election: true
I20260928 21:12:03.850313 1944 raft_consensus.cc:2454] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 0.
I20260928 21:12:03.850520 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde12d022d454fa49464f1a7439e58cb" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3747b3c6a5a049a59cd068f96b2c7068" is_pre_election: true
I20260928 21:12:03.850690 1737 leader_election.cc:302] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d239c7a25bd14cb885d4ef1c98682d62, e24fa22fc3704ac1856006a5c08cda7e; no voters:
I20260928 21:12:03.850749 1664 raft_consensus.cc:2454] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 0.
I20260928 21:12:03.850989 2019 raft_consensus.cc:2781] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:03.851049 2019 raft_consensus.cc:482] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:03.851092 2019 raft_consensus.cc:3037] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:03.852162 2019 raft_consensus.cc:504] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.852540 2019 leader_election.cc:288] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 election: Requested vote from peers 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585), d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:03.852912 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde12d022d454fa49464f1a7439e58cb" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d239c7a25bd14cb885d4ef1c98682d62"
I20260928 21:12:03.853015 1944 raft_consensus.cc:3037] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:03.852986 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dde12d022d454fa49464f1a7439e58cb" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3747b3c6a5a049a59cd068f96b2c7068"
I20260928 21:12:03.853098 1664 raft_consensus.cc:3037] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:03.854206 1664 raft_consensus.cc:2454] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 1.
I20260928 21:12:03.854206 1944 raft_consensus.cc:2454] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 1.
I20260928 21:12:03.854511 1736 leader_election.cc:302] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3747b3c6a5a049a59cd068f96b2c7068, e24fa22fc3704ac1856006a5c08cda7e; no voters:
I20260928 21:12:03.854697 2019 raft_consensus.cc:2781] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:03.854979 2019 raft_consensus.cc:686] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 1 LEADER]: Becoming Leader. State: Replica: e24fa22fc3704ac1856006a5c08cda7e, State: Running, Role: LEADER
I20260928 21:12:03.855237 2019 consensus_queue.cc:237] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.857522 1507 catalog_manager.cc:5495] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e reported cstate change: term changed from 0 to 1, leader changed from <none> to e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:12:03 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:03.904011 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.894484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45549 (local address 127.0.0.1:54494)
0928 21:12:03.894899 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:03.894912 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:03.895015 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:03.895309 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:03.895311 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:03.895682 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:03.895779 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.895785 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.897219 (+ 1434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.897222 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.897766 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:03.897773 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.898365 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.898367 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:03.898428 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.899095 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:03.899107 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:03.900500 (+ 1393us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:03.902486 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.902495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.902502 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.902726 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.903026 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:03.903028 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:03.903030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:03.903109 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:03.903553 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:03.903557 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:03.903655 (+ 98us) client_negotiation.cc:769] Sending connection context
0928 21:12:03.903744 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:12:03.903850 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"thread_start_us":138,"threads_started":1}
I20260928 21:12:03.904157 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:03.894591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54494 (local address 127.0.98.62:45549)
0928 21:12:03.894882 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:03.894891 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:03.894989 (+ 98us) server_negotiation.cc:407] Connection header received
0928 21:12:03.895110 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:03.895113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:03.895161 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:03.895281 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:03.895897 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.897112 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.897882 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:03.898246 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:03.898347 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:03.900650 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:03.900654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:03.900657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:03.900681 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:03.902376 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.902823 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.902828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.902830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.902921 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:03.903202 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:03.903205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:03.903207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:03.903446 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:03.903550 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:03.903823 (+ 273us) server_negotiation.cc:299] Negotiation successful
0928 21:12:03.903944 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":108,"threads_started":1}
I20260928 21:12:03.908182 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54494:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:12:03.909051 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:03.914054 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 7a24034cf2d449e9aec1b07d11248937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc0fa1ed4dac44b7a21b399f40305f1b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:03.914435 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a24034cf2d449e9aec1b07d11248937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.914322 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 7a24034cf2d449e9aec1b07d11248937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc0fa1ed4dac44b7a21b399f40305f1b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:03.914682 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a24034cf2d449e9aec1b07d11248937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.916896 2017 tablet_bootstrap.cc:492] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: Bootstrap starting.
I20260928 21:12:03.917670 2013 tablet_bootstrap.cc:492] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: Bootstrap starting.
I20260928 21:12:03.917577 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 7a24034cf2d449e9aec1b07d11248937 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=cc0fa1ed4dac44b7a21b399f40305f1b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:03.917927 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7a24034cf2d449e9aec1b07d11248937. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:03.918146 2017 tablet_bootstrap.cc:654] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.919068 2013 tablet_bootstrap.cc:654] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.920378 2013 tablet_bootstrap.cc:492] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: No bootstrap required, opened a new log
I20260928 21:12:03.920473 2013 ts_tablet_manager.cc:1397] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:03.920606 2014 tablet_bootstrap.cc:492] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: Bootstrap starting.
I20260928 21:12:03.920957 2013 raft_consensus.cc:348] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.921074 2013 raft_consensus.cc:374] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.921108 2013 raft_consensus.cc:729] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d239c7a25bd14cb885d4ef1c98682d62, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.921277 2013 consensus_queue.cc:260] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.921600 2013 ts_tablet_manager.cc:1428] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:03.921635 2014 tablet_bootstrap.cc:654] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:03.921562 2018 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:03.921702 2018 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:03.923178 2017 tablet_bootstrap.cc:492] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: No bootstrap required, opened a new log
I20260928 21:12:03.923269 2017 ts_tablet_manager.cc:1397] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260928 21:12:03.923787 2017 raft_consensus.cc:348] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.923900 2017 raft_consensus.cc:374] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.923935 2017 raft_consensus.cc:729] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3747b3c6a5a049a59cd068f96b2c7068, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.924072 2017 consensus_queue.cc:260] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.924371 2017 ts_tablet_manager.cc:1428] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:03.924338 2034 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:03.924516 2034 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:03.925405 2014 tablet_bootstrap.cc:492] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: No bootstrap required, opened a new log
I20260928 21:12:03.925509 2014 ts_tablet_manager.cc:1397] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 21:12:03.925904 2014 raft_consensus.cc:348] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.926009 2014 raft_consensus.cc:374] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:03.926038 2014 raft_consensus.cc:729] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e24fa22fc3704ac1856006a5c08cda7e, State: Initialized, Role: FOLLOWER
I20260928 21:12:03.926149 2014 consensus_queue.cc:260] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:03.926376 2014 ts_tablet_manager.cc:1428] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:03.926620 2019 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:03.926736 2019 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:04.156243 2019 raft_consensus.cc:482] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:04.156383 2019 raft_consensus.cc:504] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:04.156955 2019 leader_election.cc:288] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585), d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:04.157366 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a24034cf2d449e9aec1b07d11248937" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3747b3c6a5a049a59cd068f96b2c7068" is_pre_election: true
I20260928 21:12:04.157433 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a24034cf2d449e9aec1b07d11248937" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d239c7a25bd14cb885d4ef1c98682d62" is_pre_election: true
I20260928 21:12:04.157543 1664 raft_consensus.cc:2454] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 0.
I20260928 21:12:04.157653 1944 raft_consensus.cc:2454] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 0.
I20260928 21:12:04.157915 1736 leader_election.cc:302] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3747b3c6a5a049a59cd068f96b2c7068, e24fa22fc3704ac1856006a5c08cda7e; no voters:
I20260928 21:12:04.158095 2019 raft_consensus.cc:2781] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:04.158155 2019 raft_consensus.cc:482] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:04.158195 2019 raft_consensus.cc:3037] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:04.159121 2019 raft_consensus.cc:504] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:04.159463 2019 leader_election.cc:288] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 election: Requested vote from peers 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585), d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181)
I20260928 21:12:04.159777 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a24034cf2d449e9aec1b07d11248937" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3747b3c6a5a049a59cd068f96b2c7068"
I20260928 21:12:04.159904 1664 raft_consensus.cc:3037] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:04.159898 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7a24034cf2d449e9aec1b07d11248937" candidate_uuid: "e24fa22fc3704ac1856006a5c08cda7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d239c7a25bd14cb885d4ef1c98682d62"
I20260928 21:12:04.160002 1944 raft_consensus.cc:3037] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:04.161144 1944 raft_consensus.cc:2454] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 1.
I20260928 21:12:04.161144 1664 raft_consensus.cc:2454] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e24fa22fc3704ac1856006a5c08cda7e in term 1.
I20260928 21:12:04.161458 1737 leader_election.cc:302] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d239c7a25bd14cb885d4ef1c98682d62, e24fa22fc3704ac1856006a5c08cda7e; no voters:
I20260928 21:12:04.161700 2019 raft_consensus.cc:2781] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:04.161818 2019 raft_consensus.cc:686] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [term 1 LEADER]: Becoming Leader. State: Replica: e24fa22fc3704ac1856006a5c08cda7e, State: Running, Role: LEADER
I20260928 21:12:04.161972 2019 consensus_queue.cc:237] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } }
I20260928 21:12:04.162740 2046 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: TxnStatusTablet state changed. Reason: New leader e24fa22fc3704ac1856006a5c08cda7e. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.162860 2046 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:04.163136 2059 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:12:04.163311 2059 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:12:04.163897 1507 catalog_manager.cc:5495] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e reported cstate change: term changed from 0 to 1, leader changed from <none> to e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:04.227655 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.223365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49768 (local address 127.0.98.2:40245)
0928 21:12:04.223605 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:04.223608 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:04.223620 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:04.223685 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:04.223687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:04.223753 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:04.223929 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:04.224193 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.224768 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.226124 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.226353 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.226462 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.226766 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:04.227224 (+ 458us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:04.227359 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:04.227427 (+ 68us) server_negotiation.cc:299] Negotiation successful
0928 21:12:04.227480 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":101,"threads_started":1}
I20260928 21:12:04.227632 2054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.222894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40245 (local address 127.0.0.1:49768)
0928 21:12:04.223071 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:04.223086 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:04.223189 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:04.223956 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:04.223959 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:04.223967 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:04.224082 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:04.224087 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.224925 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.224928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:04.225980 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:04.225991 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.226464 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.226468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:04.226562 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.226615 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:04.227333 (+ 718us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:04.227349 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:12:04.227418 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:12:04.227501 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 21:12:04.230722 1944 raft_consensus.cc:1261] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Refusing update from remote peer e24fa22fc3704ac1856006a5c08cda7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:04.230880 1664 raft_consensus.cc:1261] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Refusing update from remote peer e24fa22fc3704ac1856006a5c08cda7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:04.231356 2019 consensus_queue.cc:1035] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:04.231531 2046 consensus_queue.cc:1035] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:04.233219 2018 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: TxnStatusTablet state changed. Reason: New leader e24fa22fc3704ac1856006a5c08cda7e. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.233333 2018 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:04.233343 2034 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: TxnStatusTablet state changed. Reason: New leader e24fa22fc3704ac1856006a5c08cda7e. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.233454 2034 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:04.235675 2019 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.235793 2019 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:04.236680 2027 mvcc.cc:205] Tried to move back new op lower bound from 7334420169643913216 to 7334420169368674304. Current Snapshot: MvccSnapshot[applied={T|T < 7334420169643913216}]
I20260928 21:12:04.237253 2034 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.237357 2034 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P 3747b3c6a5a049a59cd068f96b2c7068: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:04.240163 2046 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.240271 2046 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P e24fa22fc3704ac1856006a5c08cda7e: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:04.241386 2018 tablet_replica.cc:439] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e24fa22fc3704ac1856006a5c08cda7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 } } peers { permanent_uuid: "e24fa22fc3704ac1856006a5c08cda7e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40245 } } peers { permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 } } }
I20260928 21:12:04.241496 2018 tablet_replica.cc:442] T 7a24034cf2d449e9aec1b07d11248937 P d239c7a25bd14cb885d4ef1c98682d62: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:04.246455 2029 mvcc.cc:205] Tried to move back new op lower bound from 7334420169643913216 to 7334420169368674304. Current Snapshot: MvccSnapshot[applied={T|T < 7334420169643913216 or (T in {7334420169643913216})}]
I20260928 21:12:04.246658 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.242336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49774 (local address 127.0.98.2:40245)
0928 21:12:04.242639 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:04.242644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:04.242658 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:04.242698 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:04.242701 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:04.242747 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:04.243109 (+ 362us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:04.243195 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.243949 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.245189 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.245546 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.245914 (+ 368us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.245981 (+ 67us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:04.246074 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:04.246446 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:04.246480 (+ 34us) server_negotiation.cc:299] Negotiation successful
0928 21:12:04.246533 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260928 21:12:04.263784 2062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.254079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47785 (local address 127.0.98.2:40245)
0928 21:12:04.254189 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:04.254193 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:04.260749 (+ 6556us) server_negotiation.cc:407] Connection header received
0928 21:12:04.260964 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:04.260967 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:04.261009 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:04.261100 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:04.261331 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.261838 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.262699 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.262927 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.263036 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.263295 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:04.263380 (+ 85us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:04.263457 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:04.263544 (+ 87us) server_negotiation.cc:299] Negotiation successful
0928 21:12:04.263631 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":36}
I20260928 21:12:04.263831 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.254011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40245 (local address 127.0.98.2:47785)
0928 21:12:04.260607 (+ 6596us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:04.260626 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:04.260818 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:04.261118 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:04.261120 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:04.261127 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:04.261222 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:04.261230 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.261948 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.261954 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:04.262561 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:04.262569 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.263041 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.263045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:04.263126 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.263168 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:04.263469 (+ 301us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:04.263476 (+ 7us) client_negotiation.cc:769] Sending connection context
0928 21:12:04.263536 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:12:04.263717 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6473,"mutex_wait_us":12,"thread_start_us":96,"threads_started":1}
I20260928 21:12:04.271435 1664 raft_consensus.cc:1261] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Refusing update from remote peer e24fa22fc3704ac1856006a5c08cda7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:04.271481 1944 raft_consensus.cc:1261] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Refusing update from remote peer e24fa22fc3704ac1856006a5c08cda7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:04.271838 2068 consensus_queue.cc:1035] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "3747b3c6a5a049a59cd068f96b2c7068" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:04.272015 2046 consensus_queue.cc:1035] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d239c7a25bd14cb885d4ef1c98682d62" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:04.315341 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:04.315521 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37594:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:04.317554 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60ee874dc4554325a28ce85bfabf99d3: Sending DeleteTablet for 3 replicas of tablet dde12d022d454fa49464f1a7439e58cb
I20260928 21:12:04.318148 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet dde12d022d454fa49464f1a7439e58cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35416
I20260928 21:12:04.318322 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet dde12d022d454fa49464f1a7439e58cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:58022
I20260928 21:12:04.318342 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet dde12d022d454fa49464f1a7439e58cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38526
I20260928 21:12:04.318667 2073 tablet_replica.cc:330] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: stopping tablet replica
I20260928 21:12:04.318789 2073 raft_consensus.cc:2229] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:04.318904 2073 raft_consensus.cc:2258] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:04.319512 2073 ts_tablet_manager.cc:1905] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:04.319685 2075 tablet_replica.cc:330] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: stopping tablet replica
I20260928 21:12:04.319823 2075 raft_consensus.cc:2229] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:04.320123 2075 raft_consensus.cc:2258] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:04.320742 2075 ts_tablet_manager.cc:1905] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:04.322508 2073 ts_tablet_manager.cc:1918] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:04.322602 2073 log.cc:1196] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/dde12d022d454fa49464f1a7439e58cb
I20260928 21:12:04.322852 2073 ts_tablet_manager.cc:1939] T dde12d022d454fa49464f1a7439e58cb P 3747b3c6a5a049a59cd068f96b2c7068: Deleting consensus metadata
I20260928 21:12:04.323246 2074 tablet_replica.cc:330] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: stopping tablet replica
I20260928 21:12:04.323390 2074 raft_consensus.cc:2229] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:04.323527 2074 raft_consensus.cc:2258] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:04.323660 1492 catalog_manager.cc:4841] TS 3747b3c6a5a049a59cd068f96b2c7068 (127.0.98.1:34585): tablet dde12d022d454fa49464f1a7439e58cb (table test-table [id=e275d244a8824aaeb1fefa36ac9207c7]) successfully deleted
I20260928 21:12:04.323819 2075 ts_tablet_manager.cc:1918] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:04.323894 2075 log.cc:1196] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/dde12d022d454fa49464f1a7439e58cb
I20260928 21:12:04.324142 2075 ts_tablet_manager.cc:1939] T dde12d022d454fa49464f1a7439e58cb P e24fa22fc3704ac1856006a5c08cda7e: Deleting consensus metadata
I20260928 21:12:04.324783 1491 catalog_manager.cc:4841] TS e24fa22fc3704ac1856006a5c08cda7e (127.0.98.2:40245): tablet dde12d022d454fa49464f1a7439e58cb (table test-table [id=e275d244a8824aaeb1fefa36ac9207c7]) successfully deleted
I20260928 21:12:04.325227 2074 ts_tablet_manager.cc:1905] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:04.327296 2074 ts_tablet_manager.cc:1918] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:04.327392 2074 log.cc:1196] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/dde12d022d454fa49464f1a7439e58cb
I20260928 21:12:04.327654 2074 ts_tablet_manager.cc:1939] T dde12d022d454fa49464f1a7439e58cb P d239c7a25bd14cb885d4ef1c98682d62: Deleting consensus metadata
I20260928 21:12:04.328320 1493 catalog_manager.cc:4841] TS d239c7a25bd14cb885d4ef1c98682d62 (127.0.98.3:39181): tablet dde12d022d454fa49464f1a7439e58cb (table test-table [id=e275d244a8824aaeb1fefa36ac9207c7]) successfully deleted
Sep 28 21:12:04 dist-test-slave-v7xz krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629923, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:04.338716 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.319877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.0.98.1:34585)
0928 21:12:04.320010 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:04.320015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:04.327789 (+ 7774us) server_negotiation.cc:407] Connection header received
0928 21:12:04.327962 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:04.327966 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:04.328021 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:04.328113 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:04.329090 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.329617 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.331059 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.331283 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.331359 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.335080 (+ 3721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:04.335095 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:04.335120 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:04.335145 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:04.336933 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:04.337383 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:04.337390 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:04.337391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:04.337471 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:04.337769 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:04.337772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:04.337774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:04.338025 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:04.338242 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:04.338419 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:12:04.338543 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 21:12:04.339154 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712
I20260928 21:12:04.349562 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:04.341012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.98.62:45549)
0928 21:12:04.341167 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:04.341172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:04.341187 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:04.341312 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:04.341315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:04.341367 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:04.341458 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:04.342052 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.342939 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.343721 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:04.344108 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:04.344228 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:04.345728 (+ 1500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:04.345732 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:04.345736 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:04.345769 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:04.347738 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:04.348140 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:04.348144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:04.348146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:04.348244 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:04.348501 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:04.348504 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:04.348507 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:04.348769 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:04.348969 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:04.349206 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 21:12:04.349364 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 21:12:04.350055 1507 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54508
I20260928 21:12:04.351531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1579
I20260928 21:12:04.359750 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1719
I20260928 21:12:04.366685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1859
I20260928 21:12:04.372118 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 1475
2026-09-28T21:12:04Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3540 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:04 dist-test-slave-v7xz krb5kdc[2082](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:04 dist-test-slave-v7xz krb5kdc[2082](info): set up 2 sockets
Sep 28 21:12:04 dist-test-slave-v7xz krb5kdc[2082](info): commencing operation
krb5kdc: starting...
W20260928 21:12:05.476548 392 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:05 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:05.596405 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:32889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:05.707412 2095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:05.707742 2095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:05.707831 2095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:05.711959 2095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:05.712064 2095 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:12:05.712095 2095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:05.712157 2095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:05.712216 2095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:05.712253 2095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:05.717753 2095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:32889
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:32889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:05.718995 2095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:05.720067 2095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:05.728538 2101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:05.728596 2100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:05.728549 2103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:05.728603 2095 server_base.cc:1034] running on GCE node
I20260928 21:12:05.729329 2095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:05.730525 2095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:05.731710 2095 hybrid_clock.cc:648] HybridClock initialized: now 1790629925731696 us; error 28 us; skew 500 ppm
Sep 28 21:12:05 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:05.734515 2095 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:05.735380 2095 webserver.cc:460] Webserver started at http://127.0.98.62:46485/ using document root <none> and password file <none>
I20260928 21:12:05.735945 2095 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:05.736037 2095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:05.736277 2095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:05.738350 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "05304a08b639487a8957e8fd7074f54f"
format_stamp: "Formatted at 2026-09-28 21:12:05 on dist-test-slave-v7xz"
server_key: "2b753b21f33c7f2de887d97b277774d9"
server_key_iv: "7d103d48ae6734ddb1ee2ee20d5a2f9b"
server_key_version: "encryptionkey@0"
I20260928 21:12:05.738890 2095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "05304a08b639487a8957e8fd7074f54f"
format_stamp: "Formatted at 2026-09-28 21:12:05 on dist-test-slave-v7xz"
server_key: "2b753b21f33c7f2de887d97b277774d9"
server_key_iv: "7d103d48ae6734ddb1ee2ee20d5a2f9b"
server_key_version: "encryptionkey@0"
I20260928 21:12:05.742668 2095 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:05.745306 2110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:05.746652 2095 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:05.746820 2095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "05304a08b639487a8957e8fd7074f54f"
format_stamp: "Formatted at 2026-09-28 21:12:05 on dist-test-slave-v7xz"
server_key: "2b753b21f33c7f2de887d97b277774d9"
server_key_iv: "7d103d48ae6734ddb1ee2ee20d5a2f9b"
server_key_version: "encryptionkey@0"
I20260928 21:12:05.746954 2095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:05.757100 2095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:05.759969 2095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:05.760179 2095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:05.768209 2095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:32889
I20260928 21:12:05.768227 2172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:32889 every 8 connection(s)
I20260928 21:12:05.769276 2095 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:05.772034 2173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:05.772737 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 2095
I20260928 21:12:05.772887 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:05.773171 392 external_mini_cluster.cc:1428] Setting key 015f110bd9165507c2adf3510d5d5ef3
I20260928 21:12:05.778636 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Bootstrap starting.
I20260928 21:12:05.781658 2173 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:05.782655 2173 log.cc:824] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:12:05 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:05.785133 2173 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: No bootstrap required, opened a new log
I20260928 21:12:05.788956 2173 raft_consensus.cc:348] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05304a08b639487a8957e8fd7074f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 21:12:05.789196 2173 raft_consensus.cc:374] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:05.789265 2173 raft_consensus.cc:729] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05304a08b639487a8957e8fd7074f54f, State: Initialized, Role: FOLLOWER
I20260928 21:12:05.789456 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:05.774661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34418 (local address 127.0.98.62:32889)
0928 21:12:05.775098 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:05.775114 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:05.775151 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:12:05.775781 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:05.775799 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:05.776153 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:05.776446 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:05.777450 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:05.778750 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:05.780003 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:05.780497 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:05.780706 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:05.783856 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:05.783862 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:05.783877 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:05.783902 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:05.786257 (+ 2355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:05.786821 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:05.786827 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:05.786832 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:05.786956 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:05.787298 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:05.787300 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:05.787302 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:05.787705 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:05.787864 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:05.788171 (+ 307us) server_negotiation.cc:299] Negotiation successful
0928 21:12:05.788503 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":148,"threads_started":1}
I20260928 21:12:05.789769 2173 consensus_queue.cc:260] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05304a08b639487a8957e8fd7074f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 21:12:05.789942 2173 raft_consensus.cc:388] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:05.790019 2173 raft_consensus.cc:482] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:05.790141 2173 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:05.791092 2173 raft_consensus.cc:504] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05304a08b639487a8957e8fd7074f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 21:12:05.791451 2173 leader_election.cc:302] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 05304a08b639487a8957e8fd7074f54f; no voters:
I20260928 21:12:05.791823 2173 leader_election.cc:288] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:05.791893 2178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:05.792120 2178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [term 1 LEADER]: Becoming Leader. State: Replica: 05304a08b639487a8957e8fd7074f54f, State: Running, Role: LEADER
I20260928 21:12:05.792418 2178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05304a08b639487a8957e8fd7074f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } }
I20260928 21:12:05.793200 2173 sys_catalog.cc:564] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:05.794256 2179 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "05304a08b639487a8957e8fd7074f54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05304a08b639487a8957e8fd7074f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260928 21:12:05.794888 2179 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:05.794775 2180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 05304a08b639487a8957e8fd7074f54f. Latest consensus state: current_term: 1 leader_uuid: "05304a08b639487a8957e8fd7074f54f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "05304a08b639487a8957e8fd7074f54f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 32889 } } }
I20260928 21:12:05.795202 2180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f [sys.catalog]: This master's current role is: LEADER
W20260928 21:12:05.798004 2193 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:12:05.798128 2193 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:12:05.798234 2192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:05.799132 2192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:05.805099 2192 catalog_manager.cc:1347] Generated new cluster ID: ee81591b66404deab276a8895f7f81b1
I20260928 21:12:05.805168 2192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:05.820611 2192 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:05.821791 2192 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:05.827694 2192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Generated new TSK 0
I20260928 21:12:05.828423 2192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:05.857826 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:05.970613 2199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:05.970930 2199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:05.971014 2199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:05.975096 2199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:05.975185 2199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:05.975265 2199 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:12:05.975308 2199 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:05.975370 2199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:05.980654 2199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:05.981766 2199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:05.982788 2199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:05.992738 2207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:05.992743 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:05.992938 2199 server_base.cc:1034] running on GCE node
W20260928 21:12:05.992743 2205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:05.993597 2199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:05.994422 2199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:05.995702 2199 hybrid_clock.cc:648] HybridClock initialized: now 1790629925995664 us; error 48 us; skew 500 ppm
Sep 28 21:12:05 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:05.998543 2199 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:05.999409 2199 webserver.cc:460] Webserver started at http://127.0.98.1:37009/ using document root <none> and password file <none>
I20260928 21:12:05.999930 2199 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:05.999987 2199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:06.000145 2199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:06.002122 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "6f124857447c4f359057de8622fbe844"
format_stamp: "Formatted at 2026-09-28 21:12:05 on dist-test-slave-v7xz"
server_key: "00a1b6fef5d483f901d77c533a58016c"
server_key_iv: "ddc05dd3e795c172e2e8bf43bfc0bc1c"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.002630 2199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6f124857447c4f359057de8622fbe844"
format_stamp: "Formatted at 2026-09-28 21:12:05 on dist-test-slave-v7xz"
server_key: "00a1b6fef5d483f901d77c533a58016c"
server_key_iv: "ddc05dd3e795c172e2e8bf43bfc0bc1c"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.006213 2199 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:06.008548 2214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:06.009872 2199 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:06.010010 2199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "6f124857447c4f359057de8622fbe844"
format_stamp: "Formatted at 2026-09-28 21:12:05 on dist-test-slave-v7xz"
server_key: "00a1b6fef5d483f901d77c533a58016c"
server_key_iv: "ddc05dd3e795c172e2e8bf43bfc0bc1c"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.010114 2199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:06.037571 2199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:06.040794 2199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:06.041033 2199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:06.042569 2199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:06.042626 2199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:06.042693 2199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:06.042726 2199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:06.054003 2199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45039
I20260928 21:12:06.054080 2329 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45039 every 8 connection(s)
I20260928 21:12:06.054962 2199 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:06.057942 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.043548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40635 (local address 127.0.98.62:32889)
0928 21:12:06.043720 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.043726 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.044800 (+ 1074us) server_negotiation.cc:407] Connection header received
0928 21:12:06.046242 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.046247 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.046315 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.046408 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.048106 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.049048 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.050232 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.050552 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.050676 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.053903 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.053906 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.053908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.053933 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.055839 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.056359 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.056364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.056366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.056456 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.056802 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.056807 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.056808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.057028 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.057154 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.057629 (+ 475us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.057761 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:12:06.058698 2223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.044057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.1:40635)
0928 21:12:06.044604 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.044667 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.045944 (+ 1277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.046639 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.046648 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.047185 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:06.047953 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.047977 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.049192 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.049195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.050100 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.050109 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.050723 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.050728 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.050971 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.051614 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:06.051633 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:06.053664 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:06.055974 (+ 2310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.055980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.055994 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.056250 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.056558 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.056561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.056563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.056702 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.057154 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:06.057162 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:06.057397 (+ 235us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.057577 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.057915 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":144,"threads_started":1}
I20260928 21:12:06.060015 2330 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260928 21:12:06.060187 2330 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:06.060693 2330 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260928 21:12:06.062286 2126 ts_manager.cc:194] Registered new tserver with Master: 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039)
I20260928 21:12:06.063422 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40635
I20260928 21:12:06.064858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 2199
I20260928 21:12:06.065007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:06.065342 392 external_mini_cluster.cc:1428] Setting key 2a8b9cd4dffea9d32bfd567910722b46
I20260928 21:12:06.069625 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.062267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40141 (local address 127.0.98.62:32889)
0928 21:12:06.062401 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.062406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.062638 (+ 232us) server_negotiation.cc:407] Connection header received
0928 21:12:06.062756 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.062759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.062806 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.062889 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.063316 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.063919 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.064888 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.065190 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.065298 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.066501 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.066505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.066507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.066586 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.067993 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.068409 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.068413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.068415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.068501 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.068786 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.068791 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.068793 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.068996 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.069107 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.069360 (+ 253us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.069470 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:12:06.069638 2336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.062186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.1:40141)
0928 21:12:06.062555 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.062567 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.062678 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.062904 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.062907 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.063113 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:06.063206 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.063211 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.064049 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.064053 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.064731 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.064742 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.065335 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.065338 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.065437 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.066015 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:06.066041 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:06.066369 (+ 328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:06.068114 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.068117 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.068119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.068311 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.068610 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.068612 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.068614 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.068699 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.069098 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:06.069101 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:06.069198 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.069302 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.069440 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":289,"thread_start_us":101,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:06.091782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:06.207113 2339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:06.207554 2339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:06.207690 2339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:06.214777 2339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:06.214854 2339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:06.214897 2339 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:12:06.214912 2339 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:06.214946 2339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:06.220890 2339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2339
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:06.222101 2339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:06.223495 2339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:06.234514 2344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:06.234602 2345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:06.234603 2347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:06.234707 2339 server_base.cc:1034] running on GCE node
I20260928 21:12:06.235651 2339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:06.236326 2339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:06.237525 2339 hybrid_clock.cc:648] HybridClock initialized: now 1790629926237507 us; error 34 us; skew 500 ppm
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:06.244954 2339 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:06.245992 2339 webserver.cc:460] Webserver started at http://127.0.98.2:46359/ using document root <none> and password file <none>
I20260928 21:12:06.246594 2339 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:06.246690 2339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:06.246910 2339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:06.249071 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "930b5aea1e894c98a466832d8a19b142"
format_stamp: "Formatted at 2026-09-28 21:12:06 on dist-test-slave-v7xz"
server_key: "8b38a1e0aad4422c92daa1e70f0d13d2"
server_key_iv: "12313f10c493aedffe3cd232f0c807d0"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.249640 2339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "930b5aea1e894c98a466832d8a19b142"
format_stamp: "Formatted at 2026-09-28 21:12:06 on dist-test-slave-v7xz"
server_key: "8b38a1e0aad4422c92daa1e70f0d13d2"
server_key_iv: "12313f10c493aedffe3cd232f0c807d0"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.253453 2339 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:12:06.256125 2354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:06.257509 2339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:06.257645 2339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "930b5aea1e894c98a466832d8a19b142"
format_stamp: "Formatted at 2026-09-28 21:12:06 on dist-test-slave-v7xz"
server_key: "8b38a1e0aad4422c92daa1e70f0d13d2"
server_key_iv: "12313f10c493aedffe3cd232f0c807d0"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.257771 2339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:06.277210 2339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:06.280565 2339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:06.280790 2339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:06.282385 2339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:06.282461 2339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:06.282536 2339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:06.282578 2339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:06.293681 2339 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44947
I20260928 21:12:06.294034 2469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44947 every 8 connection(s)
I20260928 21:12:06.294807 2339 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:06.297077 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.283284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43615 (local address 127.0.98.62:32889)
0928 21:12:06.283439 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.283445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.284448 (+ 1003us) server_negotiation.cc:407] Connection header received
0928 21:12:06.285626 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.285632 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.285704 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.285819 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.287246 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.287917 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.289507 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.289803 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.289958 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.292922 (+ 2964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.292926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.292929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.292959 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.294651 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.295260 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.295264 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.295267 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.295378 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.295844 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.295847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.295849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.296086 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.296203 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.296748 (+ 545us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.296899 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:12:06.297894 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.283797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.2:43615)
0928 21:12:06.284272 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.284321 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.285360 (+ 1039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.286055 (+ 695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.286067 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.286494 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:06.287074 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.287095 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.288063 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.288066 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.289369 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.289378 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.289998 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.290002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.290203 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.290909 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:06.290929 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:06.292687 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:06.294775 (+ 2088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.294781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.294796 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.295135 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.295461 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.295466 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.295468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.295668 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.296200 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:06.296208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:06.296500 (+ 292us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.296694 (+ 194us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.297038 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":135,"threads_started":1}
I20260928 21:12:06.298866 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 2339
I20260928 21:12:06.298974 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:06.299248 2470 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260928 21:12:06.299299 392 external_mini_cluster.cc:1428] Setting key a1128bca80fe6806b8f08bcd252739f8
I20260928 21:12:06.299392 2470 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:06.299966 2470 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260928 21:12:06.301093 2126 ts_manager.cc:194] Registered new tserver with Master: 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947)
I20260928 21:12:06.301894 2126 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43615
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:06.311089 2176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.301705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56681 (local address 127.0.98.62:32889)
0928 21:12:06.302217 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.302222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.302235 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:06.302272 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.302275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.302315 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.302397 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.302983 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.303734 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.305087 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.305532 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.305678 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.307276 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.307283 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.307289 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.307322 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.309283 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.309832 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.309834 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.309835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.309911 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.310204 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.310206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.310207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.310487 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.310569 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.310869 (+ 300us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.310971 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443}
I20260928 21:12:06.311096 2476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.301626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.2:56681)
0928 21:12:06.302066 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.302085 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.302187 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.302432 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.302438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.302688 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:06.302846 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.302856 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.303884 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.303887 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.304954 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.304969 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.305693 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.305700 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.305833 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.306541 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:06.306562 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:06.307112 (+ 550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:06.309399 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.309404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.309407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.309715 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.310004 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.310007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.310009 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.310115 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.310600 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:06.310605 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:06.310710 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.310810 (+ 100us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.310931 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":223,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260928 21:12:06.717270 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.416s user 0.003s sys 0.001s
I20260928 21:12:06.717816 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:32889
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:06.836493 2479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:06.836820 2479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:06.836922 2479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:06.841063 2479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:06.841154 2479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:06.841231 2479 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260928 21:12:06.841297 2479 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:06.841406 2479 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:06.846976 2479 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:32889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2479
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:06.848112 2479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:06.849269 2479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:06.858858 2484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:06.858858 2487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:06.859007 2479 server_base.cc:1034] running on GCE node
W20260928 21:12:06.859197 2485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:06.859571 2479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:06.860275 2479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:06.861428 2479 hybrid_clock.cc:648] HybridClock initialized: now 1790629926861403 us; error 37 us; skew 500 ppm
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:06.864284 2479 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:06.865150 2479 webserver.cc:460] Webserver started at http://127.0.98.3:46297/ using document root <none> and password file <none>
I20260928 21:12:06.865674 2479 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:06.865731 2479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:06.865890 2479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:06.867805 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "f65990c38a804f5e8962e3f6593a02c1"
format_stamp: "Formatted at 2026-09-28 21:12:06 on dist-test-slave-v7xz"
server_key: "ed78855dd724ef7b954adb3ed20b98fd"
server_key_iv: "16d76ab06821384a0c543769bc6b8c02"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.868299 2479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f65990c38a804f5e8962e3f6593a02c1"
format_stamp: "Formatted at 2026-09-28 21:12:06 on dist-test-slave-v7xz"
server_key: "ed78855dd724ef7b954adb3ed20b98fd"
server_key_iv: "16d76ab06821384a0c543769bc6b8c02"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.871978 2479 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:12:06.874518 2494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:06.875775 2479 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:06.875874 2479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "f65990c38a804f5e8962e3f6593a02c1"
format_stamp: "Formatted at 2026-09-28 21:12:06 on dist-test-slave-v7xz"
server_key: "ed78855dd724ef7b954adb3ed20b98fd"
server_key_iv: "16d76ab06821384a0c543769bc6b8c02"
server_key_version: "encryptionkey@0"
I20260928 21:12:06.875960 2479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:06.887748 2479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:06.891047 2479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:06.891196 2479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:06.892498 2479 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:06.892582 2479 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:06.892666 2479 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:06.892702 2479 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:06.904083 2479 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38751
I20260928 21:12:06.904196 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38751 every 8 connection(s)
I20260928 21:12:06.905228 2479 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:06.907083 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.893426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47155 (local address 127.0.98.62:32889)
0928 21:12:06.893712 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.893719 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.894663 (+ 944us) server_negotiation.cc:407] Connection header received
0928 21:12:06.895974 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.895977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.896033 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.896149 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.897489 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.898034 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.899380 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.899629 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.899745 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.902907 (+ 3162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.902912 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.902914 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.902943 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.904873 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.905399 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.905404 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.905406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.905500 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.905927 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.905930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.905932 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.906154 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.906267 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.906758 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.906899 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":63,"threads_started":1}
I20260928 21:12:06.907836 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.893902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.3:47155)
0928 21:12:06.894450 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.894554 (+ 104us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.895704 (+ 1150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.896329 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.896339 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.896803 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:06.897340 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.897354 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.898162 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.898166 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.899255 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.899265 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.899801 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.899806 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.900030 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.900729 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:06.900747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:06.902692 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:06.905015 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.905020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.905032 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.905306 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.905615 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.905618 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.905620 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.905810 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.906282 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:06.906290 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:06.906528 (+ 238us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.906717 (+ 189us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.907029 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":142,"threads_started":1}
I20260928 21:12:06.909118 2611 heartbeater.cc:347] Connected to a master server at 127.0.98.62:32889
I20260928 21:12:06.909272 2611 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:06.909768 2611 heartbeater.cc:511] Master 127.0.98.62:32889 requested a full tablet report, sending...
I20260928 21:12:06.910827 2127 ts_manager.cc:194] Registered new tserver with Master: f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:06.911584 2127 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47155
I20260928 21:12:06.914355 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 2479
I20260928 21:12:06.914477 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:06.914762 392 external_mini_cluster.cc:1428] Setting key c752af77fd0ec551bf60f114f821b2d7
I20260928 21:12:06.916896 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:12:06.919207 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.911323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35899 (local address 127.0.98.62:32889)
0928 21:12:06.911426 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.911430 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.911705 (+ 275us) server_negotiation.cc:407] Connection header received
0928 21:12:06.911764 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.911772 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.911833 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.911922 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.912538 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.913157 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.914051 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.914307 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.914440 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.915791 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.915795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.915800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.915827 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.917525 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.917959 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.917966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.917970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.918071 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.918327 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.918333 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.918338 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.918578 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.918681 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.918932 (+ 251us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.919053 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:12:06.919215 2617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.911221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.98.3:35899)
0928 21:12:06.911572 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.911586 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.911689 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.912143 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.912145 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.912323 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:06.912418 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.912423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.913288 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.913292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.913928 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.913936 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.914420 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.914423 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.914514 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.915182 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:06.915193 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:06.915661 (+ 468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:06.917631 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.917634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.917636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.917854 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.918163 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:06.918165 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:06.918166 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:06.918238 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:06.918668 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:06.918670 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:06.918744 (+ 74us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.918875 (+ 131us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.919017 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":67,"threads_started":1}
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:12:06 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629926, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:06.944571 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.935501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34430 (local address 127.0.98.62:32889)
0928 21:12:06.935648 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.935655 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.935676 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:12:06.935724 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.935727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.935796 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.935889 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:06.936623 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.937457 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.938611 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.938856 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.938975 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.941643 (+ 2668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:06.941647 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:06.941649 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:06.941676 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:06.942962 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.943424 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.943427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.943428 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.943506 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:06.943802 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:06.943806 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:06.943807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:06.944011 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:06.944133 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.944329 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.944440 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:12:06.947649 2127 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:06.950819 2127 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:06.965277 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.959739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55316 (local address 127.0.98.1:45039)
0928 21:12:06.960361 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.960370 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.960644 (+ 274us) server_negotiation.cc:407] Connection header received
0928 21:12:06.960729 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.960739 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.961044 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.961217 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:06.961536 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.962567 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.964025 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.964563 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.964688 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.964837 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.965041 (+ 204us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.965142 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":98,"threads_started":1}
I20260928 21:12:06.965291 2630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.959569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45039 (local address 127.0.0.1:55316)
0928 21:12:06.960291 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.960341 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.960503 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.961214 (+ 711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.961219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.961240 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:06.961371 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.961377 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.962675 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.962680 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.963900 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.963908 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.964684 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.964688 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.964789 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.964824 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.965008 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.965130 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":1,"thread_start_us":140,"threads_started":1}
I20260928 21:12:06.966203 2629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.960369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38751 (local address 127.0.0.1:40990)
0928 21:12:06.960997 (+ 628us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.961010 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.961090 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.961627 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.961630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.961648 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:06.961757 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.961774 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.963178 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.963182 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.964625 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.964631 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.965804 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.965808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.965912 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.965936 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.966010 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.966078 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"mutex_wait_us":155,"thread_start_us":96,"threads_started":1}
I20260928 21:12:06.966324 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.960636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40990 (local address 127.0.98.3:38751)
0928 21:12:06.961140 (+ 504us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.961152 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.961171 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:12:06.961239 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.961243 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.961476 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.961640 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:06.961890 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.963041 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.964756 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.965590 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.965690 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.965827 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.966063 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.966171 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":114,"threads_started":1}
I20260928 21:12:06.966408 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.959819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56830 (local address 127.0.98.2:44947)
0928 21:12:06.960174 (+ 355us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:06.960181 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:06.960944 (+ 763us) server_negotiation.cc:407] Connection header received
0928 21:12:06.961241 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:06.961247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:06.961442 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:06.961554 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:06.962026 (+ 472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.963105 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.964756 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.965590 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.965677 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.965814 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:06.966187 (+ 373us) server_negotiation.cc:299] Negotiation successful
0928 21:12:06.966273 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":77,"threads_started":1}
I20260928 21:12:06.966789 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:06.959637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44947 (local address 127.0.0.1:56830)
0928 21:12:06.960687 (+ 1050us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:06.960700 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:06.960794 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:06.961749 (+ 955us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:06.961752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:06.961769 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:06.961878 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.961884 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.963214 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.963216 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.964615 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:06.964626 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:06.965925 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:06.965929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:06.966039 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:06.966061 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:12:06.966566 (+ 505us) client_negotiation.cc:240] Negotiation successful
0928 21:12:06.966665 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":992,"mutex_wait_us":115,"thread_start_us":90,"threads_started":1}
I20260928 21:12:06.969117 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6077cb3e44c84ac5c15f6612659c9 (DEFAULT_TABLE table=test-table [id=27da3503653d4da69cbd7a2768474a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:06.968942 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6077cb3e44c84ac5c15f6612659c9 (DEFAULT_TABLE table=test-table [id=27da3503653d4da69cbd7a2768474a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:06.969456 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 31f6077cb3e44c84ac5c15f6612659c9 (DEFAULT_TABLE table=test-table [id=27da3503653d4da69cbd7a2768474a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:06.970049 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6077cb3e44c84ac5c15f6612659c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:06.970055 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6077cb3e44c84ac5c15f6612659c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:06.970310 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f6077cb3e44c84ac5c15f6612659c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:06.976212 2635 tablet_bootstrap.cc:492] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Bootstrap starting.
I20260928 21:12:06.977849 2637 tablet_bootstrap.cc:492] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Bootstrap starting.
I20260928 21:12:06.978922 2635 tablet_bootstrap.cc:654] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:06.979717 2637 tablet_bootstrap.cc:654] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:06.979717 2635 log.cc:824] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:06.980310 2636 tablet_bootstrap.cc:492] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Bootstrap starting.
I20260928 21:12:06.980509 2637 log.cc:824] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:06.981854 2635 tablet_bootstrap.cc:492] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: No bootstrap required, opened a new log
I20260928 21:12:06.982055 2635 ts_tablet_manager.cc:1397] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:12:06.982277 2636 tablet_bootstrap.cc:654] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:06.982556 2637 tablet_bootstrap.cc:492] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: No bootstrap required, opened a new log
I20260928 21:12:06.982745 2637 ts_tablet_manager.cc:1397] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20260928 21:12:06.983014 2636 log.cc:824] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:06.984918 2636 tablet_bootstrap.cc:492] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: No bootstrap required, opened a new log
I20260928 21:12:06.985100 2636 ts_tablet_manager.cc:1397] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:12:06.986271 2635 raft_consensus.cc:348] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:06.986515 2635 raft_consensus.cc:374] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:06.986574 2635 raft_consensus.cc:729] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Initialized, Role: FOLLOWER
I20260928 21:12:06.986790 2637 raft_consensus.cc:348] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:06.987042 2637 raft_consensus.cc:374] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:06.987107 2637 raft_consensus.cc:729] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65990c38a804f5e8962e3f6593a02c1, State: Initialized, Role: FOLLOWER
I20260928 21:12:06.987033 2635 consensus_queue.cc:260] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:06.987525 2637 consensus_queue.cc:260] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:06.988107 2635 ts_tablet_manager.cc:1428] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:12:06.988232 2611 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260928 21:12:06.988485 2637 ts_tablet_manager.cc:1428] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260928 21:12:06.988499 2470 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260928 21:12:06.989342 2636 raft_consensus.cc:348] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:06.989578 2636 raft_consensus.cc:374] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:06.989847 2636 raft_consensus.cc:729] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f124857447c4f359057de8622fbe844, State: Initialized, Role: FOLLOWER
I20260928 21:12:06.990324 2636 consensus_queue.cc:260] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:06.991000 2330 heartbeater.cc:503] Master 127.0.98.62:32889 was elected leader, sending a full tablet report...
I20260928 21:12:06.992058 2636 ts_tablet_manager.cc:1428] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260928 21:12:07.046802 2471 tablet.cc:2369] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:07.056576 2331 tablet.cc:2369] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:07.156297 2612 tablet.cc:2369] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:07.311019 2641 raft_consensus.cc:482] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:07.311209 2641 raft_consensus.cc:504] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.312295 2641 leader_election.cc:288] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:07.316857 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.312675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34375 (local address 127.0.98.3:38751)
0928 21:12:07.312846 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.312852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.313115 (+ 263us) server_negotiation.cc:407] Connection header received
0928 21:12:07.313287 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.313291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.313342 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.313427 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:07.313696 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.314494 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.315838 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.316253 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.316334 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.316421 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.316610 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.316694 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:12:07.316831 2645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.312545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38751 (local address 127.0.98.2:34375)
0928 21:12:07.312996 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.313009 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.313164 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.313438 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.313440 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.313455 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:07.313588 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.313593 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.314631 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.314634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.315711 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.315720 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.316381 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.316385 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.316466 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.316496 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.316568 (+ 72us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.316654 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"mutex_wait_us":63,"thread_start_us":134,"threads_started":1}
I20260928 21:12:07.316913 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.312667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34295 (local address 127.0.98.1:45039)
0928 21:12:07.312811 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.312815 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.313191 (+ 376us) server_negotiation.cc:407] Connection header received
0928 21:12:07.313350 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.313353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.313395 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.313480 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:07.313811 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.314538 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.315874 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.316267 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.316395 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.316457 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.316672 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.316752 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:12:07.316874 2644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.312545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45039 (local address 127.0.98.2:34295)
0928 21:12:07.313083 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.313094 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.313229 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.313487 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.313490 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.313505 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:07.313636 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.313641 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.314632 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.314634 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.315720 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.315729 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.316418 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.316422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.316513 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.316529 (+ 16us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.316641 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.316702 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":207,"thread_start_us":168,"threads_started":1}
I20260928 21:12:07.317622 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6077cb3e44c84ac5c15f6612659c9" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1" is_pre_election: true
I20260928 21:12:07.317929 2565 raft_consensus.cc:2454] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 0.
I20260928 21:12:07.318467 2357 leader_election.cc:302] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930b5aea1e894c98a466832d8a19b142, f65990c38a804f5e8962e3f6593a02c1; no voters:
I20260928 21:12:07.318526 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6077cb3e44c84ac5c15f6612659c9" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f124857447c4f359057de8622fbe844" is_pre_election: true
I20260928 21:12:07.318794 2284 raft_consensus.cc:2454] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 0.
I20260928 21:12:07.318807 2641 raft_consensus.cc:2781] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:07.318879 2641 raft_consensus.cc:482] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:07.318949 2641 raft_consensus.cc:3037] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.320024 2641 raft_consensus.cc:504] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.320398 2641 leader_election.cc:288] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 election: Requested vote from peers 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:07.320739 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6077cb3e44c84ac5c15f6612659c9" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1"
I20260928 21:12:07.320770 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f6077cb3e44c84ac5c15f6612659c9" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f124857447c4f359057de8622fbe844"
I20260928 21:12:07.320907 2565 raft_consensus.cc:3037] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.320933 2284 raft_consensus.cc:3037] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.322259 2284 raft_consensus.cc:2454] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 1.
I20260928 21:12:07.322567 2355 leader_election.cc:302] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, 930b5aea1e894c98a466832d8a19b142; no voters:
I20260928 21:12:07.322636 2565 raft_consensus.cc:2454] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 1.
I20260928 21:12:07.322736 2641 raft_consensus.cc:2781] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:07.322975 2641 raft_consensus.cc:686] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 1 LEADER]: Becoming Leader. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Running, Role: LEADER
I20260928 21:12:07.323287 2641 consensus_queue.cc:237] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.326761 2126 catalog_manager.cc:5495] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 reported cstate change: term changed from 0 to 1, leader changed from <none> to 930b5aea1e894c98a466832d8a19b142 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "930b5aea1e894c98a466832d8a19b142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } health_report { overall_health: UNKNOWN } } }
Sep 28 21:12:07 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629925, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:07.443251 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.435054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:32889 (local address 127.0.0.1:34434)
0928 21:12:07.435372 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.435387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.435499 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.435739 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.435741 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.436156 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:07.436260 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.436266 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.437053 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.437057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.437703 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.437712 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.438122 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.438125 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.438201 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.438834 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:07.438846 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:07.440265 (+ 1419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:07.441883 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:07.441890 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:07.441895 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:07.442137 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:07.442374 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:07.442376 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:07.442377 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:07.442456 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:07.442784 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:07.442789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:07.442902 (+ 113us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.442994 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.443117 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":137,"threads_started":1}
I20260928 21:12:07.443292 2503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.434923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34434 (local address 127.0.98.62:32889)
0928 21:12:07.435136 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.435143 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.435467 (+ 324us) server_negotiation.cc:407] Connection header received
0928 21:12:07.435586 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.435588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.435630 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.435704 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:07.436369 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.436961 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.437805 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.438005 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.438107 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.440402 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:07.440405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:07.440407 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:07.440428 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:07.441788 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:07.442220 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:07.442223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:07.442224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:07.442292 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:07.442532 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:07.442534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:07.442535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:07.442695 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:07.442768 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.443045 (+ 277us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.443183 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":122}
I20260928 21:12:07.447185 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34434:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:12:07.447952 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:07.452749 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 682419955bae481590e2b4a05f4655ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d82320f813045adb3d159bfd7d2b294]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:07.452914 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 682419955bae481590e2b4a05f4655ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d82320f813045adb3d159bfd7d2b294]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:07.453524 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682419955bae481590e2b4a05f4655ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:07.453524 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682419955bae481590e2b4a05f4655ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:07.453181 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 682419955bae481590e2b4a05f4655ba (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8d82320f813045adb3d159bfd7d2b294]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:07.453895 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 682419955bae481590e2b4a05f4655ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:07.456296 2635 tablet_bootstrap.cc:492] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: Bootstrap starting.
I20260928 21:12:07.456377 2636 tablet_bootstrap.cc:492] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: Bootstrap starting.
I20260928 21:12:07.457283 2636 tablet_bootstrap.cc:654] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:07.457290 2635 tablet_bootstrap.cc:654] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:07.457835 2637 tablet_bootstrap.cc:492] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: Bootstrap starting.
I20260928 21:12:07.458628 2636 tablet_bootstrap.cc:492] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: No bootstrap required, opened a new log
I20260928 21:12:07.458694 2637 tablet_bootstrap.cc:654] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:07.458742 2636 ts_tablet_manager.cc:1397] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:07.459245 2636 raft_consensus.cc:348] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.459380 2636 raft_consensus.cc:374] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:07.459420 2636 raft_consensus.cc:729] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f124857447c4f359057de8622fbe844, State: Initialized, Role: FOLLOWER
I20260928 21:12:07.459571 2636 consensus_queue.cc:260] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.459642 2635 tablet_bootstrap.cc:492] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: No bootstrap required, opened a new log
I20260928 21:12:07.459748 2635 ts_tablet_manager.cc:1397] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:12:07.459872 2636 ts_tablet_manager.cc:1428] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:07.460135 2637 tablet_bootstrap.cc:492] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: No bootstrap required, opened a new log
I20260928 21:12:07.460208 2637 ts_tablet_manager.cc:1397] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260928 21:12:07.460215 2635 raft_consensus.cc:348] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.460364 2635 raft_consensus.cc:374] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:07.460294 2643 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.460404 2635 raft_consensus.cc:729] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Initialized, Role: FOLLOWER
I20260928 21:12:07.460435 2643 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.460548 2635 consensus_queue.cc:260] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.460693 2637 raft_consensus.cc:348] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.460794 2637 raft_consensus.cc:374] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:07.460855 2635 ts_tablet_manager.cc:1428] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 21:12:07.460865 2637 raft_consensus.cc:729] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65990c38a804f5e8962e3f6593a02c1, State: Initialized, Role: FOLLOWER
I20260928 21:12:07.460793 2641 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.460975 2641 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.460992 2637 consensus_queue.cc:260] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.461254 2637 ts_tablet_manager.cc:1428] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:12:07.461916 2642 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.462049 2642 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.522182 2643 raft_consensus.cc:482] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:07.522348 2643 raft_consensus.cc:504] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.523512 2643 leader_election.cc:288] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:07.527973 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.523856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39329 (local address 127.0.98.2:44947)
0928 21:12:07.524078 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.524081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.524117 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:12:07.524273 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.524277 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.524340 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.524448 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:07.524775 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.525537 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.526743 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.527355 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.527461 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.527529 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.527743 (+ 214us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.527830 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260928 21:12:07.527992 2657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.523790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44947 (local address 127.0.98.1:39329)
0928 21:12:07.524029 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.524052 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.524151 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.524479 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.524483 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.524505 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:07.524651 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.524658 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.525638 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.525642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.526623 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.526632 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.527498 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.527502 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.527618 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.527643 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.527717 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.527801 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":84,"threads_started":1}
I20260928 21:12:07.528038 2659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.524154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38751 (local address 127.0.98.1:39903)
0928 21:12:07.524499 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.524513 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.524651 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.525084 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.525087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.525104 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:07.525219 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.525225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.526088 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.526091 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.526952 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.526961 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.527639 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.527642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.527747 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.527771 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.527845 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.527910 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":109,"threads_started":1}
I20260928 21:12:07.528075 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.524260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39903 (local address 127.0.98.3:38751)
0928 21:12:07.524423 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.524428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.524619 (+ 191us) server_negotiation.cc:407] Connection header received
0928 21:12:07.524732 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.524735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.524784 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.524903 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:07.525334 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.525974 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.527085 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.527526 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.527628 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.527687 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.527883 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.527950 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:12:07.528566 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682419955bae481590e2b4a05f4655ba" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1" is_pre_election: true
I20260928 21:12:07.528728 2565 raft_consensus.cc:2454] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 0.
I20260928 21:12:07.528776 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682419955bae481590e2b4a05f4655ba" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930b5aea1e894c98a466832d8a19b142" is_pre_election: true
I20260928 21:12:07.529070 2424 raft_consensus.cc:2454] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 0.
I20260928 21:12:07.529140 2217 leader_election.cc:302] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, f65990c38a804f5e8962e3f6593a02c1; no voters:
I20260928 21:12:07.529433 2643 raft_consensus.cc:2781] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:07.529526 2643 raft_consensus.cc:482] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:07.529578 2643 raft_consensus.cc:3037] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.530544 2643 raft_consensus.cc:504] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.530915 2643 leader_election.cc:288] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 election: Requested vote from peers 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:07.531296 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682419955bae481590e2b4a05f4655ba" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930b5aea1e894c98a466832d8a19b142"
I20260928 21:12:07.531332 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "682419955bae481590e2b4a05f4655ba" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1"
I20260928 21:12:07.531430 2424 raft_consensus.cc:3037] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.531452 2565 raft_consensus.cc:3037] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.532321 2424 raft_consensus.cc:2454] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 1.
I20260928 21:12:07.532372 2565 raft_consensus.cc:2454] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 1.
I20260928 21:12:07.532615 2217 leader_election.cc:302] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, 930b5aea1e894c98a466832d8a19b142; no voters:
I20260928 21:12:07.532774 2643 raft_consensus.cc:2781] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:07.533119 2643 raft_consensus.cc:686] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [term 1 LEADER]: Becoming Leader. State: Replica: 6f124857447c4f359057de8622fbe844, State: Running, Role: LEADER
I20260928 21:12:07.533488 2643 consensus_queue.cc:237] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.535588 2660 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: TxnStatusTablet state changed. Reason: New leader 6f124857447c4f359057de8622fbe844. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.535717 2660 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:07.536123 2662 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:12:07.536329 2662 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:12:07.536612 2125 catalog_manager.cc:5495] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f124857447c4f359057de8622fbe844 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:07.563443 2653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.559377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45039 (local address 127.0.0.1:55320)
0928 21:12:07.559504 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.559521 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.559645 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.560019 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.560022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.560030 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:07.560140 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.560146 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.560979 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.560984 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.561720 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.561747 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.562222 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.562226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.562320 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.562363 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.563185 (+ 822us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.563198 (+ 13us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.563258 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.563331 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:12:07.563490 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.559498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55320 (local address 127.0.98.1:45039)
0928 21:12:07.559633 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.559638 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.559653 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:07.559845 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.559847 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.559899 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.559998 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:07.560244 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.560868 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.561876 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.562110 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.562222 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.562522 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.563070 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.563189 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.563296 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.563359 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:12:07.568791 2424 raft_consensus.cc:1261] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Refusing update from remote peer 6f124857447c4f359057de8622fbe844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:07.568938 2565 raft_consensus.cc:1261] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Refusing update from remote peer 6f124857447c4f359057de8622fbe844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:07.569518 2660 consensus_queue.cc:1035] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:07.569710 2643 consensus_queue.cc:1035] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:07.572901 2641 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: TxnStatusTablet state changed. Reason: New leader 6f124857447c4f359057de8622fbe844. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.573012 2641 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.574829 2642 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: TxnStatusTablet state changed. Reason: New leader 6f124857447c4f359057de8622fbe844. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.574959 2642 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.576433 2660 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.576550 2660 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:07.577297 2667 mvcc.cc:205] Tried to move back new op lower bound from 7334420183314878464 to 7334420183180087296. Current Snapshot: MvccSnapshot[applied={T|T < 7334420183314878464}]
I20260928 21:12:07.577406 2641 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.577522 2641 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 930b5aea1e894c98a466832d8a19b142: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.578948 2660 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.579080 2660 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P 6f124857447c4f359057de8622fbe844: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:07.579825 2642 tablet_replica.cc:439] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } } }
I20260928 21:12:07.579933 2642 tablet_replica.cc:442] T 682419955bae481590e2b4a05f4655ba P f65990c38a804f5e8962e3f6593a02c1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:07.580644 2666 mvcc.cc:205] Tried to move back new op lower bound from 7334420183314878464 to 7334420183180087296. Current Snapshot: MvccSnapshot[applied={T|T < 7334420183314878464}]
I20260928 21:12:07.591040 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.586704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56838 (local address 127.0.98.2:44947)
0928 21:12:07.586841 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.586846 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.587030 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:12:07.587142 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.587145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.587215 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.587334 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:07.587518 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.588294 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.589379 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.589799 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.589912 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.590151 (+ 239us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.590673 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.590785 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.590842 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.590897 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:12:07.602423 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.598786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46861 (local address 127.0.98.1:45039)
0928 21:12:07.598907 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.598911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.599170 (+ 259us) server_negotiation.cc:407] Connection header received
0928 21:12:07.599320 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.599323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.599388 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.599500 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:07.599749 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.600234 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.601265 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.601545 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.601665 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.601960 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.602038 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.602121 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.602237 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.602287 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:07.602445 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.598695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45039 (local address 127.0.98.2:46861)
0928 21:12:07.599055 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.599067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.599219 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.599510 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.599513 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.599520 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:07.599643 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.599648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.600338 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.600345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.601142 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.601151 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.601676 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.601679 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.601769 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.601807 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.602124 (+ 317us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.602132 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.602215 (+ 83us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.602288 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":113,"threads_started":1}
I20260928 21:12:07.609326 2565 raft_consensus.cc:1261] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Refusing update from remote peer 930b5aea1e894c98a466832d8a19b142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:07.609500 2284 raft_consensus.cc:1261] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Refusing update from remote peer 930b5aea1e894c98a466832d8a19b142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:07.609798 2646 consensus_queue.cc:1035] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:07.609985 2641 consensus_queue.cc:1035] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:07.619239 2665 mvcc.cc:205] Tried to move back new op lower bound from 7334420183483256832 to 7334420182318694400. Current Snapshot: MvccSnapshot[applied={T|T < 7334420183483256832}]
I20260928 21:12:07.632092 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.627905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44947 (local address 127.0.98.1:56697)
0928 21:12:07.628270 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.628285 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.628419 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.628777 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.628779 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.628787 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:07.628920 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.628926 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.629680 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.629684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.630670 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.630682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.631375 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.631379 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.631468 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.631520 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.631836 (+ 316us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.631841 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.631892 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.631960 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":101,"threads_started":1}
I20260928 21:12:07.632087 2658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.627997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56697 (local address 127.0.98.2:44947)
0928 21:12:07.628109 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.628114 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.628348 (+ 234us) server_negotiation.cc:407] Connection header received
0928 21:12:07.628532 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.628535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.628593 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.628645 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:07.629058 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.629570 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.630953 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.631274 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.631377 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.631665 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.631754 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.631827 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.631902 (+ 75us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.631955 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:12:07.663111 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:07.663319 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:07.666257 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Sending DeleteTablet for 3 replicas of tablet 31f6077cb3e44c84ac5c15f6612659c9
I20260928 21:12:07.667241 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6077cb3e44c84ac5c15f6612659c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40990
I20260928 21:12:07.667776 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34430:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:07.668200 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:07.668503 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6077cb3e44c84ac5c15f6612659c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260928 21:12:07.668980 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f6077cb3e44c84ac5c15f6612659c9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:07 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56830
I20260928 21:12:07.669241 2688 tablet_replica.cc:330] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: stopping tablet replica
I20260928 21:12:07.669625 2688 raft_consensus.cc:2229] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:07.669911 2688 raft_consensus.cc:2258] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:07.670435 2690 tablet_replica.cc:330] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: stopping tablet replica
I20260928 21:12:07.670738 2690 raft_consensus.cc:2229] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:07.671106 2690 raft_consensus.cc:2258] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:07.671540 2689 tablet_replica.cc:330] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: stopping tablet replica
I20260928 21:12:07.672695 2688 ts_tablet_manager.cc:1905] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:07.673071 2689 raft_consensus.cc:2229] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:07.673313 2689 raft_consensus.cc:2258] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:07.673468 2690 ts_tablet_manager.cc:1905] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:07.674808 2264 tablet_service.cc:1467] Processing CreateTablet for tablet 8f01cc0032364de290570a865f922541 (DEFAULT_TABLE table=test-table [id=f02b775075984bb5b0702367c2a5daf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:07.675132 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f01cc0032364de290570a865f922541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:07.676721 2688 ts_tablet_manager.cc:1918] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:07.676908 2688 log.cc:1196] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/31f6077cb3e44c84ac5c15f6612659c9
I20260928 21:12:07.677325 2688 ts_tablet_manager.cc:1939] T 31f6077cb3e44c84ac5c15f6612659c9 P f65990c38a804f5e8962e3f6593a02c1: Deleting consensus metadata
I20260928 21:12:07.677834 2404 tablet_service.cc:1467] Processing CreateTablet for tablet 8f01cc0032364de290570a865f922541 (DEFAULT_TABLE table=test-table [id=f02b775075984bb5b0702367c2a5daf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:07.678087 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f01cc0032364de290570a865f922541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:07.678316 2533 tablet_service.cc:1467] Processing CreateTablet for tablet 8f01cc0032364de290570a865f922541 (DEFAULT_TABLE table=test-table [id=f02b775075984bb5b0702367c2a5daf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:07.678545 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f01cc0032364de290570a865f922541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:07.680385 2637 tablet_bootstrap.cc:492] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Bootstrap starting.
I20260928 21:12:07.680506 2635 tablet_bootstrap.cc:492] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Bootstrap starting.
I20260928 21:12:07.680590 2113 catalog_manager.cc:4841] TS f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751): tablet 31f6077cb3e44c84ac5c15f6612659c9 (table test-table [id=27da3503653d4da69cbd7a2768474a01]) successfully deleted
I20260928 21:12:07.680986 2636 tablet_bootstrap.cc:492] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Bootstrap starting.
I20260928 21:12:07.681375 2637 tablet_bootstrap.cc:654] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:07.678082 2690 ts_tablet_manager.cc:1918] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:07.681494 2635 tablet_bootstrap.cc:654] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:07.681479 2690 log.cc:1196] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/31f6077cb3e44c84ac5c15f6612659c9
I20260928 21:12:07.681855 2690 ts_tablet_manager.cc:1939] T 31f6077cb3e44c84ac5c15f6612659c9 P 930b5aea1e894c98a466832d8a19b142: Deleting consensus metadata
I20260928 21:12:07.682039 2636 tablet_bootstrap.cc:654] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:07.682773 2637 tablet_bootstrap.cc:492] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: No bootstrap required, opened a new log
I20260928 21:12:07.682854 2637 ts_tablet_manager.cc:1397] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260928 21:12:07.682920 2113 catalog_manager.cc:4841] TS 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947): tablet 31f6077cb3e44c84ac5c15f6612659c9 (table test-table [id=27da3503653d4da69cbd7a2768474a01]) successfully deleted
I20260928 21:12:07.683290 2637 raft_consensus.cc:348] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.683394 2637 raft_consensus.cc:374] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:07.683423 2637 raft_consensus.cc:729] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65990c38a804f5e8962e3f6593a02c1, State: Initialized, Role: FOLLOWER
I20260928 21:12:07.683547 2637 consensus_queue.cc:260] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.683787 2637 ts_tablet_manager.cc:1428] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:12:07.683897 2689 ts_tablet_manager.cc:1905] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:07.686797 2689 ts_tablet_manager.cc:1918] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:07.686919 2689 log.cc:1196] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/31f6077cb3e44c84ac5c15f6612659c9
I20260928 21:12:07.687234 2689 ts_tablet_manager.cc:1939] T 31f6077cb3e44c84ac5c15f6612659c9 P 6f124857447c4f359057de8622fbe844: Deleting consensus metadata
I20260928 21:12:07.688258 2111 catalog_manager.cc:4841] TS 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039): tablet 31f6077cb3e44c84ac5c15f6612659c9 (table test-table [id=27da3503653d4da69cbd7a2768474a01]) successfully deleted
I20260928 21:12:07.693715 2636 tablet_bootstrap.cc:492] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: No bootstrap required, opened a new log
I20260928 21:12:07.693812 2636 ts_tablet_manager.cc:1397] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.000s
I20260928 21:12:07.694334 2636 raft_consensus.cc:348] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.694446 2636 raft_consensus.cc:374] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:07.694444 2635 tablet_bootstrap.cc:492] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: No bootstrap required, opened a new log
I20260928 21:12:07.694478 2636 raft_consensus.cc:729] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f124857447c4f359057de8622fbe844, State: Initialized, Role: FOLLOWER
I20260928 21:12:07.694559 2635 ts_tablet_manager.cc:1397] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.000s
I20260928 21:12:07.694595 2636 consensus_queue.cc:260] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.694850 2636 ts_tablet_manager.cc:1428] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:07.694958 2635 raft_consensus.cc:348] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.695063 2635 raft_consensus.cc:374] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:07.695091 2635 raft_consensus.cc:729] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Initialized, Role: FOLLOWER
I20260928 21:12:07.695196 2635 consensus_queue.cc:260] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.695420 2635 ts_tablet_manager.cc:1428] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:07.800241 2643 raft_consensus.cc:482] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:07.800395 2643 raft_consensus.cc:504] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.800891 2643 leader_election.cc:288] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:07.801415 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f01cc0032364de290570a865f922541" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930b5aea1e894c98a466832d8a19b142" is_pre_election: true
I20260928 21:12:07.801468 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f01cc0032364de290570a865f922541" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1" is_pre_election: true
I20260928 21:12:07.801595 2565 raft_consensus.cc:2454] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 0.
I20260928 21:12:07.801645 2424 raft_consensus.cc:2454] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 0.
I20260928 21:12:07.801970 2217 leader_election.cc:302] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, 930b5aea1e894c98a466832d8a19b142; no voters:
I20260928 21:12:07.802310 2643 raft_consensus.cc:2781] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:07.802383 2643 raft_consensus.cc:482] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:07.802425 2643 raft_consensus.cc:3037] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.803723 2643 raft_consensus.cc:504] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.804100 2643 leader_election.cc:288] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 election: Requested vote from peers 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:07.804416 2424 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f01cc0032364de290570a865f922541" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "930b5aea1e894c98a466832d8a19b142"
I20260928 21:12:07.804559 2424 raft_consensus.cc:3037] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.804554 2565 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f01cc0032364de290570a865f922541" candidate_uuid: "6f124857447c4f359057de8622fbe844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1"
I20260928 21:12:07.804677 2565 raft_consensus.cc:3037] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:07.805791 2424 raft_consensus.cc:2454] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 1.
I20260928 21:12:07.805810 2565 raft_consensus.cc:2454] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f124857447c4f359057de8622fbe844 in term 1.
I20260928 21:12:07.806077 2217 leader_election.cc:302] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, f65990c38a804f5e8962e3f6593a02c1; no voters:
I20260928 21:12:07.806289 2643 raft_consensus.cc:2781] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:07.806380 2643 raft_consensus.cc:686] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 1 LEADER]: Becoming Leader. State: Replica: 6f124857447c4f359057de8622fbe844, State: Running, Role: LEADER
I20260928 21:12:07.806520 2643 consensus_queue.cc:237] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:07.808440 2125 catalog_manager.cc:5495] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f124857447c4f359057de8622fbe844 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "6f124857447c4f359057de8622fbe844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:07.840211 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.836982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55328 (local address 127.0.98.1:45039)
0928 21:12:07.837101 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.837105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.837157 (+ 52us) server_negotiation.cc:407] Connection header received
0928 21:12:07.837281 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.837284 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.837336 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.837417 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:07.837619 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.838393 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.839186 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.839466 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.839569 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.839790 (+ 221us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.839867 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.839935 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.840014 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.840067 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:12:07.848639 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.845931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45039 (local address 127.0.98.1:53881)
0928 21:12:07.846082 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:07.846096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:07.846162 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:07.846352 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:07.846354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:07.846359 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:07.846439 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.846443 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.847109 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.847111 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.847655 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:07.847661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.848081 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.848083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:07.848141 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.848164 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.848406 (+ 242us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.848410 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:12:07.848459 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:12:07.848524 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:12:07.848639 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:07.846035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53881 (local address 127.0.98.1:45039)
0928 21:12:07.846144 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:07.846148 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:07.846160 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:07.846226 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:07.846228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:07.846264 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:07.846330 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:07.846524 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.847000 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.847773 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:07.847977 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:07.848066 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:07.848270 (+ 204us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:07.848331 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:07.848385 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:07.848479 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:12:07.848515 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:12:07.854259 2565 raft_consensus.cc:1261] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Refusing update from remote peer 6f124857447c4f359057de8622fbe844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:07.854321 2424 raft_consensus.cc:1261] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Refusing update from remote peer 6f124857447c4f359057de8622fbe844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:07.854759 2643 consensus_queue.cc:1035] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:07.854879 2660 consensus_queue.cc:1035] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:12:07.868721 2682 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:12:09.423753 2126 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:09.423918 2126 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34430:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:09.426529 2126 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Sending DeleteTablet for 3 replicas of tablet 8f01cc0032364de290570a865f922541
I20260928 21:12:09.427146 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f01cc0032364de290570a865f922541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260928 21:12:09.427249 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f01cc0032364de290570a865f922541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40990
I20260928 21:12:09.427482 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f01cc0032364de290570a865f922541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56830
I20260928 21:12:09.427593 2720 tablet_replica.cc:330] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: stopping tablet replica
I20260928 21:12:09.427772 2720 raft_consensus.cc:2229] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:09.427919 2720 raft_consensus.cc:2258] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:09.428363 2721 tablet_replica.cc:330] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: stopping tablet replica
I20260928 21:12:09.428568 2720 ts_tablet_manager.cc:1905] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:09.428565 2721 raft_consensus.cc:2229] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:09.428783 2721 raft_consensus.cc:2258] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:09.429325 2719 tablet_replica.cc:330] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: stopping tablet replica
I20260928 21:12:09.429625 2719 raft_consensus.cc:2229] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:09.429986 2719 raft_consensus.cc:2258] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:09.430364 2721 ts_tablet_manager.cc:1905] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:09.430888 2719 ts_tablet_manager.cc:1905] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:09.431689 2720 ts_tablet_manager.cc:1918] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:12:09.431771 2720 log.cc:1196] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/8f01cc0032364de290570a865f922541
I20260928 21:12:09.432034 2720 ts_tablet_manager.cc:1939] T 8f01cc0032364de290570a865f922541 P f65990c38a804f5e8962e3f6593a02c1: Deleting consensus metadata
I20260928 21:12:09.432821 2113 catalog_manager.cc:4841] TS f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751): tablet 8f01cc0032364de290570a865f922541 (table test-table [id=f02b775075984bb5b0702367c2a5daf9]) successfully deleted
I20260928 21:12:09.433086 2721 ts_tablet_manager.cc:1918] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:12:09.433182 2721 log.cc:1196] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/8f01cc0032364de290570a865f922541
I20260928 21:12:09.433418 2721 ts_tablet_manager.cc:1939] T 8f01cc0032364de290570a865f922541 P 930b5aea1e894c98a466832d8a19b142: Deleting consensus metadata
I20260928 21:12:09.434005 2113 catalog_manager.cc:4841] TS 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947): tablet 8f01cc0032364de290570a865f922541 (table test-table [id=f02b775075984bb5b0702367c2a5daf9]) successfully deleted
I20260928 21:12:09.434237 2719 ts_tablet_manager.cc:1918] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:12:09.434320 2719 log.cc:1196] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/8f01cc0032364de290570a865f922541
I20260928 21:12:09.434643 2719 ts_tablet_manager.cc:1939] T 8f01cc0032364de290570a865f922541 P 6f124857447c4f359057de8622fbe844: Deleting consensus metadata
I20260928 21:12:09.435441 2111 catalog_manager.cc:4841] TS 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039): tablet 8f01cc0032364de290570a865f922541 (table test-table [id=f02b775075984bb5b0702367c2a5daf9]) successfully deleted
Sep 28 21:12:09 dist-test-slave-v7xz krb5kdc[2082](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:12:09 dist-test-slave-v7xz krb5kdc[2082](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629929, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:09.456697 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:09.447978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34442 (local address 127.0.98.62:32889)
0928 21:12:09.448255 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:09.448261 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:09.448324 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:12:09.448424 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:09.448426 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:09.448469 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:09.448553 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:09.449322 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:09.449892 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:09.450898 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:09.451118 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:09.451200 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:09.453640 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:09.453643 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:09.453645 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:09.453670 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:09.455217 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:09.455591 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:09.455593 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:09.455595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:09.455668 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:09.455883 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:09.455885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:09.455886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:09.456055 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:09.456154 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:09.456465 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 21:12:09.456564 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":124,"threads_started":1}
I20260928 21:12:09.459488 2126 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:09.460012 2126 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:09.464936 2404 tablet_service.cc:1467] Processing CreateTablet for tablet a1309d64f7fd4c81919b5f8e55d6c0cc (DEFAULT_TABLE table=test-table [id=ba3733815d3b4d68b2907289d2ed72ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:09.464983 2533 tablet_service.cc:1467] Processing CreateTablet for tablet a1309d64f7fd4c81919b5f8e55d6c0cc (DEFAULT_TABLE table=test-table [id=ba3733815d3b4d68b2907289d2ed72ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:09.464984 2264 tablet_service.cc:1467] Processing CreateTablet for tablet a1309d64f7fd4c81919b5f8e55d6c0cc (DEFAULT_TABLE table=test-table [id=ba3733815d3b4d68b2907289d2ed72ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:09.465247 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1309d64f7fd4c81919b5f8e55d6c0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:09.465247 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1309d64f7fd4c81919b5f8e55d6c0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:09.465291 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a1309d64f7fd4c81919b5f8e55d6c0cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:09.467839 2730 tablet_bootstrap.cc:492] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Bootstrap starting.
I20260928 21:12:09.467849 2731 tablet_bootstrap.cc:492] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Bootstrap starting.
I20260928 21:12:09.468339 2732 tablet_bootstrap.cc:492] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Bootstrap starting.
I20260928 21:12:09.468818 2730 tablet_bootstrap.cc:654] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:09.469245 2731 tablet_bootstrap.cc:654] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:09.469310 2732 tablet_bootstrap.cc:654] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:09.470114 2730 tablet_bootstrap.cc:492] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: No bootstrap required, opened a new log
I20260928 21:12:09.470237 2730 ts_tablet_manager.cc:1397] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:09.470434 2731 tablet_bootstrap.cc:492] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: No bootstrap required, opened a new log
I20260928 21:12:09.470531 2731 ts_tablet_manager.cc:1397] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:09.470630 2732 tablet_bootstrap.cc:492] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: No bootstrap required, opened a new log
I20260928 21:12:09.470706 2732 ts_tablet_manager.cc:1397] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260928 21:12:09.470681 2730 raft_consensus.cc:348] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.470795 2730 raft_consensus.cc:374] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:09.470836 2730 raft_consensus.cc:729] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f124857447c4f359057de8622fbe844, State: Initialized, Role: FOLLOWER
I20260928 21:12:09.470973 2731 raft_consensus.cc:348] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.470999 2730 consensus_queue.cc:260] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.471084 2731 raft_consensus.cc:374] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:09.471074 2732 raft_consensus.cc:348] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.471200 2732 raft_consensus.cc:374] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:09.471246 2732 raft_consensus.cc:729] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Initialized, Role: FOLLOWER
I20260928 21:12:09.471338 2730 ts_tablet_manager.cc:1428] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:09.471124 2731 raft_consensus.cc:729] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65990c38a804f5e8962e3f6593a02c1, State: Initialized, Role: FOLLOWER
I20260928 21:12:09.471395 2732 consensus_queue.cc:260] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.471566 2731 consensus_queue.cc:260] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.471949 2731 ts_tablet_manager.cc:1428] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:09.471835 2732 ts_tablet_manager.cc:1428] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:12:09.840862 2736 raft_consensus.cc:482] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:09.841027 2736 raft_consensus.cc:504] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.841511 2736 leader_election.cc:288] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:09.842077 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1309d64f7fd4c81919b5f8e55d6c0cc" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f124857447c4f359057de8622fbe844" is_pre_election: true
I20260928 21:12:09.842118 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1309d64f7fd4c81919b5f8e55d6c0cc" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1" is_pre_election: true
I20260928 21:12:09.842255 2564 raft_consensus.cc:2454] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 0.
I20260928 21:12:09.842269 2284 raft_consensus.cc:2454] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 0.
I20260928 21:12:09.842643 2357 leader_election.cc:302] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, 930b5aea1e894c98a466832d8a19b142, f65990c38a804f5e8962e3f6593a02c1; no voters:
I20260928 21:12:09.842849 2736 raft_consensus.cc:2781] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:09.842921 2736 raft_consensus.cc:482] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:09.842949 2736 raft_consensus.cc:3037] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:09.844026 2736 raft_consensus.cc:504] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.844424 2736 leader_election.cc:288] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 election: Requested vote from peers 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:09.844933 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1309d64f7fd4c81919b5f8e55d6c0cc" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1"
I20260928 21:12:09.844933 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a1309d64f7fd4c81919b5f8e55d6c0cc" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f124857447c4f359057de8622fbe844"
I20260928 21:12:09.845075 2284 raft_consensus.cc:3037] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:09.845108 2564 raft_consensus.cc:3037] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:09.846336 2564 raft_consensus.cc:2454] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 1.
I20260928 21:12:09.846335 2284 raft_consensus.cc:2454] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 1.
I20260928 21:12:09.846665 2357 leader_election.cc:302] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930b5aea1e894c98a466832d8a19b142, f65990c38a804f5e8962e3f6593a02c1; no voters:
I20260928 21:12:09.846875 2736 raft_consensus.cc:2781] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:09.847150 2736 raft_consensus.cc:686] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 1 LEADER]: Becoming Leader. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Running, Role: LEADER
I20260928 21:12:09.847321 2736 consensus_queue.cc:237] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.849152 2127 catalog_manager.cc:5495] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 reported cstate change: term changed from 0 to 1, leader changed from <none> to 930b5aea1e894c98a466832d8a19b142 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "930b5aea1e894c98a466832d8a19b142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:09.929812 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:09.926713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56846 (local address 127.0.98.2:44947)
0928 21:12:09.926980 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:09.926988 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:09.927000 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:09.927040 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:09.927042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:09.927083 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:09.927156 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:09.927379 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:09.927925 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:09.928909 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:09.929132 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:09.929215 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:09.929433 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:09.929505 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:09.929559 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:09.929637 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 21:12:09.929681 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":131,"threads_started":1}
I20260928 21:12:09.936640 2284 raft_consensus.cc:1261] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Refusing update from remote peer 930b5aea1e894c98a466832d8a19b142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:09.936759 2564 raft_consensus.cc:1261] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Refusing update from remote peer 930b5aea1e894c98a466832d8a19b142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:09.937084 2736 consensus_queue.cc:1035] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:09.937263 2738 consensus_queue.cc:1035] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:09.970053 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:09.970245 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:09.972378 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Sending DeleteTablet for 3 replicas of tablet a1309d64f7fd4c81919b5f8e55d6c0cc
I20260928 21:12:09.972988 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet a1309d64f7fd4c81919b5f8e55d6c0cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56830
I20260928 21:12:09.972981 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet a1309d64f7fd4c81919b5f8e55d6c0cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260928 21:12:09.973080 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet a1309d64f7fd4c81919b5f8e55d6c0cc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40990
I20260928 21:12:09.973286 2756 tablet_replica.cc:330] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: stopping tablet replica
I20260928 21:12:09.973381 2758 tablet_replica.cc:330] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: stopping tablet replica
I20260928 21:12:09.973397 2756 raft_consensus.cc:2229] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:09.973510 2758 raft_consensus.cc:2229] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:09.973587 2756 raft_consensus.cc:2258] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:09.973667 2758 raft_consensus.cc:2258] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:09.973935 2125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:12:09.974213 2756 ts_tablet_manager.cc:1905] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:09.974345 2758 ts_tablet_manager.cc:1905] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:12:09.974370 2125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:09.973657 2757 tablet_replica.cc:330] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: stopping tablet replica
I20260928 21:12:09.975150 2757 raft_consensus.cc:2229] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:09.975257 2757 raft_consensus.cc:2258] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:09.975905 2757 ts_tablet_manager.cc:1905] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:09.976913 2756 ts_tablet_manager.cc:1918] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:09.976989 2756 log.cc:1196] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/a1309d64f7fd4c81919b5f8e55d6c0cc
I20260928 21:12:09.977290 2756 ts_tablet_manager.cc:1939] T a1309d64f7fd4c81919b5f8e55d6c0cc P 930b5aea1e894c98a466832d8a19b142: Deleting consensus metadata
I20260928 21:12:09.978065 2113 catalog_manager.cc:4841] TS 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947): tablet a1309d64f7fd4c81919b5f8e55d6c0cc (table test-table [id=ba3733815d3b4d68b2907289d2ed72ae]) successfully deleted
I20260928 21:12:09.978560 2757 ts_tablet_manager.cc:1918] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:09.978619 2758 ts_tablet_manager.cc:1918] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:09.978664 2757 log.cc:1196] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/a1309d64f7fd4c81919b5f8e55d6c0cc
I20260928 21:12:09.978734 2758 log.cc:1196] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/a1309d64f7fd4c81919b5f8e55d6c0cc
I20260928 21:12:09.978964 2757 ts_tablet_manager.cc:1939] T a1309d64f7fd4c81919b5f8e55d6c0cc P 6f124857447c4f359057de8622fbe844: Deleting consensus metadata
I20260928 21:12:09.979034 2758 ts_tablet_manager.cc:1939] T a1309d64f7fd4c81919b5f8e55d6c0cc P f65990c38a804f5e8962e3f6593a02c1: Deleting consensus metadata
I20260928 21:12:09.979755 2533 tablet_service.cc:1467] Processing CreateTablet for tablet ef71d6f163534206a3f419e20be37218 (DEFAULT_TABLE table=test-table [id=af753661118b44dcbc08cb38922d3f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:09.979729 2404 tablet_service.cc:1467] Processing CreateTablet for tablet ef71d6f163534206a3f419e20be37218 (DEFAULT_TABLE table=test-table [id=af753661118b44dcbc08cb38922d3f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:09.979995 2404 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef71d6f163534206a3f419e20be37218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:09.980011 2533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef71d6f163534206a3f419e20be37218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:09.981318 2264 tablet_service.cc:1467] Processing CreateTablet for tablet ef71d6f163534206a3f419e20be37218 (DEFAULT_TABLE table=test-table [id=af753661118b44dcbc08cb38922d3f3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:09.981552 2264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef71d6f163534206a3f419e20be37218. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:09.982108 2759 tablet_bootstrap.cc:492] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Bootstrap starting.
I20260928 21:12:09.982340 2113 catalog_manager.cc:4841] TS f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751): tablet a1309d64f7fd4c81919b5f8e55d6c0cc (table test-table [id=ba3733815d3b4d68b2907289d2ed72ae]) successfully deleted
I20260928 21:12:09.982884 2111 catalog_manager.cc:4841] TS 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039): tablet a1309d64f7fd4c81919b5f8e55d6c0cc (table test-table [id=ba3733815d3b4d68b2907289d2ed72ae]) successfully deleted
I20260928 21:12:09.982968 2759 tablet_bootstrap.cc:654] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:09.983743 2762 tablet_bootstrap.cc:492] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844: Bootstrap starting.
I20260928 21:12:09.984020 2759 tablet_bootstrap.cc:492] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: No bootstrap required, opened a new log
I20260928 21:12:09.984128 2759 ts_tablet_manager.cc:1397] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:09.984409 2760 tablet_bootstrap.cc:492] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Bootstrap starting.
I20260928 21:12:09.984570 2759 raft_consensus.cc:348] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.984678 2762 tablet_bootstrap.cc:654] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:09.984707 2759 raft_consensus.cc:374] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:09.984750 2759 raft_consensus.cc:729] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Initialized, Role: FOLLOWER
I20260928 21:12:09.984941 2759 consensus_queue.cc:260] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.985450 2760 tablet_bootstrap.cc:654] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:09.985875 2759 ts_tablet_manager.cc:1428] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:09.985988 2762 tablet_bootstrap.cc:492] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844: No bootstrap required, opened a new log
I20260928 21:12:09.986063 2762 ts_tablet_manager.cc:1397] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:09.986377 2762 raft_consensus.cc:348] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.986501 2762 raft_consensus.cc:374] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:09.986538 2762 raft_consensus.cc:729] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f124857447c4f359057de8622fbe844, State: Initialized, Role: FOLLOWER
I20260928 21:12:09.986694 2762 consensus_queue.cc:260] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.987042 2762 ts_tablet_manager.cc:1428] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:09.988466 2760 tablet_bootstrap.cc:492] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: No bootstrap required, opened a new log
I20260928 21:12:09.988575 2760 ts_tablet_manager.cc:1397] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:12:09.989066 2760 raft_consensus.cc:348] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.989203 2760 raft_consensus.cc:374] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:09.989243 2760 raft_consensus.cc:729] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f65990c38a804f5e8962e3f6593a02c1, State: Initialized, Role: FOLLOWER
I20260928 21:12:09.989380 2760 consensus_queue.cc:260] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:09.989648 2760 ts_tablet_manager.cc:1428] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:12:10.115921 2755 raft_consensus.cc:482] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:10.116061 2755 raft_consensus.cc:504] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:10.116518 2755 leader_election.cc:288] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:10.117015 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef71d6f163534206a3f419e20be37218" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f124857447c4f359057de8622fbe844" is_pre_election: true
I20260928 21:12:10.117064 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef71d6f163534206a3f419e20be37218" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1" is_pre_election: true
I20260928 21:12:10.117206 2284 raft_consensus.cc:2454] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 0.
I20260928 21:12:10.117234 2564 raft_consensus.cc:2454] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 0.
I20260928 21:12:10.117545 2355 leader_election.cc:302] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6f124857447c4f359057de8622fbe844, 930b5aea1e894c98a466832d8a19b142; no voters:
I20260928 21:12:10.117728 2755 raft_consensus.cc:2781] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:10.117800 2755 raft_consensus.cc:482] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:10.117856 2755 raft_consensus.cc:3037] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:10.118901 2755 raft_consensus.cc:504] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:10.119264 2755 leader_election.cc:288] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 election: Requested vote from peers 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039), f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751)
I20260928 21:12:10.119809 2284 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef71d6f163534206a3f419e20be37218" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f124857447c4f359057de8622fbe844"
I20260928 21:12:10.119809 2564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef71d6f163534206a3f419e20be37218" candidate_uuid: "930b5aea1e894c98a466832d8a19b142" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f65990c38a804f5e8962e3f6593a02c1"
I20260928 21:12:10.119948 2564 raft_consensus.cc:3037] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:10.119948 2284 raft_consensus.cc:3037] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:10.121169 2564 raft_consensus.cc:2454] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 1.
I20260928 21:12:10.121263 2284 raft_consensus.cc:2454] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 930b5aea1e894c98a466832d8a19b142 in term 1.
I20260928 21:12:10.121502 2357 leader_election.cc:302] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 930b5aea1e894c98a466832d8a19b142, f65990c38a804f5e8962e3f6593a02c1; no voters:
I20260928 21:12:10.121676 2755 raft_consensus.cc:2781] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:10.121766 2755 raft_consensus.cc:686] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 1 LEADER]: Becoming Leader. State: Replica: 930b5aea1e894c98a466832d8a19b142, State: Running, Role: LEADER
I20260928 21:12:10.121915 2755 consensus_queue.cc:237] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } }
I20260928 21:12:10.123739 2125 catalog_manager.cc:5495] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 reported cstate change: term changed from 0 to 1, leader changed from <none> to 930b5aea1e894c98a466832d8a19b142 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "930b5aea1e894c98a466832d8a19b142" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "930b5aea1e894c98a466832d8a19b142" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44947 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:10.148087 2284 raft_consensus.cc:1261] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Refusing update from remote peer 930b5aea1e894c98a466832d8a19b142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:10.148456 2564 raft_consensus.cc:1261] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Refusing update from remote peer 930b5aea1e894c98a466832d8a19b142: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:10.148545 2755 consensus_queue.cc:1035] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f124857447c4f359057de8622fbe844" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:10.148880 2755 consensus_queue.cc:1035] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f65990c38a804f5e8962e3f6593a02c1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:12:10.161644 2743 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260928 21:12:11.710520 2125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:11.710671 2125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:11.712966 2125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 05304a08b639487a8957e8fd7074f54f: Sending DeleteTablet for 3 replicas of tablet ef71d6f163534206a3f419e20be37218
I20260928 21:12:11.713634 2264 tablet_service.cc:1514] Processing DeleteTablet for tablet ef71d6f163534206a3f419e20be37218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55316
I20260928 21:12:11.713646 2404 tablet_service.cc:1514] Processing DeleteTablet for tablet ef71d6f163534206a3f419e20be37218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56830
I20260928 21:12:11.713701 2533 tablet_service.cc:1514] Processing DeleteTablet for tablet ef71d6f163534206a3f419e20be37218 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:11 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40990
I20260928 21:12:11.713982 2790 tablet_replica.cc:330] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844: stopping tablet replica
I20260928 21:12:11.714129 2790 raft_consensus.cc:2229] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:11.714181 2791 tablet_replica.cc:330] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: stopping tablet replica
I20260928 21:12:11.714267 2790 raft_consensus.cc:2258] T ef71d6f163534206a3f419e20be37218 P 6f124857447c4f359057de8622fbe844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:11.714335 2791 raft_consensus.cc:2229] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:11.714514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 2199
I20260928 21:12:11.714532 2791 raft_consensus.cc:2258] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:11.715745 2792 tablet_replica.cc:330] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: stopping tablet replica
I20260928 21:12:11.715895 2792 raft_consensus.cc:2229] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:11.716007 2792 raft_consensus.cc:2258] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:11.717217 2792 ts_tablet_manager.cc:1905] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:11.717237 2791 ts_tablet_manager.cc:1905] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:11.719309 2792 ts_tablet_manager.cc:1918] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:12:11.719414 2792 log.cc:1196] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/ef71d6f163534206a3f419e20be37218
I20260928 21:12:11.719660 2792 ts_tablet_manager.cc:1939] T ef71d6f163534206a3f419e20be37218 P f65990c38a804f5e8962e3f6593a02c1: Deleting consensus metadata
I20260928 21:12:11.720373 2113 catalog_manager.cc:4841] TS f65990c38a804f5e8962e3f6593a02c1 (127.0.98.3:38751): tablet ef71d6f163534206a3f419e20be37218 (table test-table [id=af753661118b44dcbc08cb38922d3f3e]) successfully deleted
I20260928 21:12:11.720598 2791 ts_tablet_manager.cc:1918] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260928 21:12:11.720680 2791 log.cc:1196] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/ef71d6f163534206a3f419e20be37218
I20260928 21:12:11.720994 2791 ts_tablet_manager.cc:1939] T ef71d6f163534206a3f419e20be37218 P 930b5aea1e894c98a466832d8a19b142: Deleting consensus metadata
I20260928 21:12:11.721747 2113 catalog_manager.cc:4841] TS 930b5aea1e894c98a466832d8a19b142 (127.0.98.2:44947): tablet ef71d6f163534206a3f419e20be37218 (table test-table [id=af753661118b44dcbc08cb38922d3f3e]) successfully deleted
W20260928 21:12:11.723726 2111 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45039 (error 108)
I20260928 21:12:11.724499 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 2339
I20260928 21:12:11.725042 2794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:11.724483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45039 (local address 127.0.0.1:55340)
0928 21:12:11.724733 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:11.724811 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45039: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
W20260928 21:12:11.725144 2111 catalog_manager.cc:4584] TS 6f124857447c4f359057de8622fbe844 (127.0.98.1:45039): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ef71d6f163534206a3f419e20be37218: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45039: connect: Connection refused (error 111)
I20260928 21:12:11.731364 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 2479
I20260928 21:12:11.737145 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 2095
2026-09-28T21:12:11Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (7365 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:11 dist-test-slave-v7xz krb5kdc[2797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:11 dist-test-slave-v7xz krb5kdc[2797](info): set up 2 sockets
Sep 28 21:12:11 dist-test-slave-v7xz krb5kdc[2797](info): commencing operation
krb5kdc: starting...
W20260928 21:12:12.788820 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:12 dist-test-slave-v7xz krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:12.908475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34367
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34367
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:13.018503 2810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:13.018841 2810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:13.018927 2810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:13.023168 2810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:13.023250 2810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:13.023288 2810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:13.023321 2810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:13.023348 2810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:13.028923 2810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34367
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:13.030056 2810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:13.031111 2810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:13.040108 2810 server_base.cc:1034] running on GCE node
W20260928 21:12:13.040203 2815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:13.040243 2818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:13.040243 2816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:13.041113 2810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:13.042232 2810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:13.043393 2810 hybrid_clock.cc:648] HybridClock initialized: now 1790629933043359 us; error 52 us; skew 500 ppm
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:13.046348 2810 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:13.047350 2810 webserver.cc:460] Webserver started at http://127.0.98.62:43535/ using document root <none> and password file <none>
I20260928 21:12:13.047912 2810 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:13.047972 2810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:13.048144 2810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:13.050365 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "b8070deeb21340a3ae461d41805c9661"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "d0bf4d19a81574fb035e29725eac65dd"
server_key_iv: "5b9bb8d7a6f1c9ea73a699d346b407bd"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.050930 2810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "b8070deeb21340a3ae461d41805c9661"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "d0bf4d19a81574fb035e29725eac65dd"
server_key_iv: "5b9bb8d7a6f1c9ea73a699d346b407bd"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.054898 2810 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:13.057382 2825 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.058684 2810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:13.058815 2810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "b8070deeb21340a3ae461d41805c9661"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "d0bf4d19a81574fb035e29725eac65dd"
server_key_iv: "5b9bb8d7a6f1c9ea73a699d346b407bd"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.058941 2810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:13.078105 2810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:13.081453 2810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:13.081661 2810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:13.088972 2810 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34367
I20260928 21:12:13.088997 2877 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34367 every 8 connection(s)
I20260928 21:12:13.089915 2810 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:13.092626 2878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:13.095018 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 2810
I20260928 21:12:13.095130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:13.095420 392 external_mini_cluster.cc:1428] Setting key fa956733823f5ed12974035874864ff7
I20260928 21:12:13.098740 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661: Bootstrap starting.
I20260928 21:12:13.100982 2878 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:13.101709 2878 log.cc:824] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:13.103490 2878 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661: No bootstrap required, opened a new log
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:13.106462 2878 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8070deeb21340a3ae461d41805c9661" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260928 21:12:13.106707 2878 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:13.106797 2878 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8070deeb21340a3ae461d41805c9661, State: Initialized, Role: FOLLOWER
I20260928 21:12:13.107264 2878 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8070deeb21340a3ae461d41805c9661" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260928 21:12:13.107414 2878 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:13.107498 2878 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:13.107597 2878 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:13.108592 2878 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8070deeb21340a3ae461d41805c9661" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260928 21:12:13.108989 2878 leader_election.cc:302] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8070deeb21340a3ae461d41805c9661; no voters:
I20260928 21:12:13.109299 2878 leader_election.cc:288] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:13.109407 2883 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:13.109617 2883 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [term 1 LEADER]: Becoming Leader. State: Replica: b8070deeb21340a3ae461d41805c9661, State: Running, Role: LEADER
I20260928 21:12:13.109915 2883 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8070deeb21340a3ae461d41805c9661" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } }
I20260928 21:12:13.110134 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.096902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37024 (local address 127.0.98.62:34367)
0928 21:12:13.097316 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:13.097331 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:13.097370 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:12:13.098027 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:13.098045 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:13.098379 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:13.098715 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:13.099513 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.100486 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.101451 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.101972 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.102150 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.104561 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:13.104566 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:13.104579 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:13.104606 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:13.106914 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.107351 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.107358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.107363 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.107524 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.107835 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.107842 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.107843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.108212 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:13.108372 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:13.108675 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 21:12:13.109090 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":139,"threads_started":1}
I20260928 21:12:13.110595 2878 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:13.111088 2883 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8070deeb21340a3ae461d41805c9661. Latest consensus state: current_term: 1 leader_uuid: "b8070deeb21340a3ae461d41805c9661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8070deeb21340a3ae461d41805c9661" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } } }
I20260928 21:12:13.111207 2883 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:13.111651 2884 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8070deeb21340a3ae461d41805c9661" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8070deeb21340a3ae461d41805c9661" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34367 } } }
I20260928 21:12:13.111817 2884 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:13.112582 2887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:13.115624 2887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:13.121603 2887 catalog_manager.cc:1347] Generated new cluster ID: aed4b417ac704b569a8b98311d4a3216
I20260928 21:12:13.121680 2887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:13.130769 2887 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:13.131713 2887 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:13.143347 2887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8070deeb21340a3ae461d41805c9661: Generated new TSK 0
I20260928 21:12:13.144104 2887 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:13.172394 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:13.295610 2904 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:13.295917 2904 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:13.295964 2904 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:13.300298 2904 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:13.300374 2904 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:13.300467 2904 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:13.306026 2904 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2904
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:13.307148 2904 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:13.308194 2904 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:13.318373 2912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:13.318377 2910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:13.318641 2904 server_base.cc:1034] running on GCE node
W20260928 21:12:13.318373 2909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:13.319149 2904 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:13.319754 2904 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:13.320932 2904 hybrid_clock.cc:648] HybridClock initialized: now 1790629933320920 us; error 30 us; skew 500 ppm
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:13.323693 2904 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:13.324541 2904 webserver.cc:460] Webserver started at http://127.0.98.1:43739/ using document root <none> and password file <none>
I20260928 21:12:13.325172 2904 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:13.325235 2904 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:13.325471 2904 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:13.327472 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "303f375c3598447e8ec87b1711cdc292"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "c87f2c2d42aeb2bc5dd51ab02c90e5a5"
server_key_iv: "a006547089ebb0281b72fa2fee286e0a"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.328032 2904 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "303f375c3598447e8ec87b1711cdc292"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "c87f2c2d42aeb2bc5dd51ab02c90e5a5"
server_key_iv: "a006547089ebb0281b72fa2fee286e0a"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.331743 2904 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260928 21:12:13.334434 2919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.335635 2904 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:13.335765 2904 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "303f375c3598447e8ec87b1711cdc292"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "c87f2c2d42aeb2bc5dd51ab02c90e5a5"
server_key_iv: "a006547089ebb0281b72fa2fee286e0a"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.335873 2904 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:13.351110 2904 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:13.354269 2904 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:13.354475 2904 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:13.355132 2904 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:13.356065 2904 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:13.356127 2904 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.356173 2904 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:13.356189 2904 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.365504 2904 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43249
I20260928 21:12:13.365525 3032 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43249 every 8 connection(s)
I20260928 21:12:13.366443 2904 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:13.369421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 2904
I20260928 21:12:13.369537 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:13.369861 392 external_mini_cluster.cc:1428] Setting key e25506076884989677ff309a06bacf8f
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:12:13.382277 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.368206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52809 (local address 127.0.98.62:34367)
0928 21:12:13.368342 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:13.368346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:13.369266 (+ 920us) server_negotiation.cc:407] Connection header received
0928 21:12:13.371006 (+ 1740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:13.371009 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:13.371063 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:13.371137 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:13.372743 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.373702 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.374870 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.375301 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.375430 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.378339 (+ 2909us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:13.378344 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:13.378347 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:13.378378 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:13.380152 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.380679 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.380682 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.380683 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.380758 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.381233 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.381236 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.381237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.381441 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:13.381525 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:13.382047 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 21:12:13.382153 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:13.383342 3035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.368611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34367 (local address 127.0.98.1:52809)
0928 21:12:13.369082 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:13.369124 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:13.370791 (+ 1667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:13.371335 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:13.371343 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:13.371839 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:13.372558 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:13.372580 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.373840 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.373843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:13.374736 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:13.374746 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.375501 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.375505 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:13.375678 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.376353 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:13.376370 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:13.378159 (+ 1789us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:13.380287 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:13.380292 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:13.380305 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:13.380579 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:13.380965 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:13.380968 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:13.380969 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:13.381102 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:13.381534 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:13.381539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:13.381797 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 21:12:13.381976 (+ 179us) client_negotiation.cc:240] Negotiation successful
0928 21:12:13.382250 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":172,"threads_started":1}
I20260928 21:12:13.384648 3033 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34367
I20260928 21:12:13.384960 3033 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:13.385504 3033 heartbeater.cc:511] Master 127.0.98.62:34367 requested a full tablet report, sending...
I20260928 21:12:13.387185 2842 ts_manager.cc:194] Registered new tserver with Master: 303f375c3598447e8ec87b1711cdc292 (127.0.98.1:43249)
I20260928 21:12:13.389019 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:13.399691 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:13.511307 3038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:13.511603 3038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:13.511775 3038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:13.515995 3038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:13.516054 3038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:13.516156 3038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:13.521396 3038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:13.522452 3038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:13.523497 3038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:13.533156 3044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:13.533144 3043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:13.533183 3038 server_base.cc:1034] running on GCE node
W20260928 21:12:13.533144 3046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:13.533780 3038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:13.534453 3038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:13.535612 3038 hybrid_clock.cc:648] HybridClock initialized: now 1790629933535576 us; error 54 us; skew 500 ppm
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:13.538470 3038 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:13.539399 3038 webserver.cc:460] Webserver started at http://127.0.98.2:32823/ using document root <none> and password file <none>
I20260928 21:12:13.539939 3038 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:13.540026 3038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:13.540208 3038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:13.542232 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "054fb6a86ab4419786d3fd55980a36ff"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "2f4b55b144d4b8081aff885ff1b2f71f"
server_key_iv: "03a4e37a8b2983caaf0a57dc3eebd6a0"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.542732 3038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "054fb6a86ab4419786d3fd55980a36ff"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "2f4b55b144d4b8081aff885ff1b2f71f"
server_key_iv: "03a4e37a8b2983caaf0a57dc3eebd6a0"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.546583 3038 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:13.549114 3053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.550414 3038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:13.550557 3038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "054fb6a86ab4419786d3fd55980a36ff"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "2f4b55b144d4b8081aff885ff1b2f71f"
server_key_iv: "03a4e37a8b2983caaf0a57dc3eebd6a0"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.550679 3038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:13.568219 3038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:13.571498 3038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:13.571709 3038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:13.572345 3038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:13.573236 3038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:13.573308 3038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.573387 3038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:13.573436 3038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.582561 3038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35939
I20260928 21:12:13.582584 3166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35939 every 8 connection(s)
I20260928 21:12:13.583472 3038 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:13.586192 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3038
I20260928 21:12:13.586321 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:13.586640 392 external_mini_cluster.cc:1428] Setting key 05617f9b6efe922230d5a275db98dd35
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:13.598852 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.585302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49201 (local address 127.0.98.62:34367)
0928 21:12:13.585460 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:13.585464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:13.586382 (+ 918us) server_negotiation.cc:407] Connection header received
0928 21:12:13.588054 (+ 1672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:13.588058 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:13.588107 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:13.588207 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:13.589903 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.590772 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.591921 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.592144 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.592259 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.595055 (+ 2796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:13.595059 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:13.595061 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:13.595084 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:13.596599 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.597217 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.597222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.597224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.597307 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.597668 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.597672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.597673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.597926 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:13.598039 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:13.598562 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 21:12:13.598687 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:13.599928 3169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.585685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34367 (local address 127.0.98.2:49201)
0928 21:12:13.586196 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:13.586259 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:13.587802 (+ 1543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:13.588431 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:13.588441 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:13.589019 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:13.589736 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:13.589758 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.590902 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.590905 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:13.591787 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:13.591796 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.592298 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.592302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:13.592480 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.593126 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:13.593143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:13.594872 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:13.596760 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:13.596766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:13.596781 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:13.597109 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:13.597408 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:13.597411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:13.597413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:13.597559 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:13.598049 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:13.598058 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:13.598319 (+ 261us) client_negotiation.cc:769] Sending connection context
0928 21:12:13.598509 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 21:12:13.598863 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260928 21:12:13.601270 3167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34367
I20260928 21:12:13.601528 3167 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:13.602037 3167 heartbeater.cc:511] Master 127.0.98.62:34367 requested a full tablet report, sending...
I20260928 21:12:13.603089 2842 ts_manager.cc:194] Registered new tserver with Master: 054fb6a86ab4419786d3fd55980a36ff (127.0.98.2:35939)
I20260928 21:12:13.603665 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49201
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:13.616602 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:13.727731 3172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:13.727974 3172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:13.728016 3172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:13.732039 3172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:13.732089 3172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:13.732146 3172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:13.737166 3172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34367
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:13.738202 3172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:13.739194 3172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:13.748773 3178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:13.748742 3177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:13.748764 3180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:13.749049 3172 server_base.cc:1034] running on GCE node
I20260928 21:12:13.749626 3172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:13.750350 3172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:13.751564 3172 hybrid_clock.cc:648] HybridClock initialized: now 1790629933751541 us; error 35 us; skew 500 ppm
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:13.754462 3172 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:13.755462 3172 webserver.cc:460] Webserver started at http://127.0.98.3:38787/ using document root <none> and password file <none>
I20260928 21:12:13.756029 3172 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:13.756116 3172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:13.756351 3172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:13.758467 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "d4456c0c033845cea16ac2695d5bf0ac"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "1c86957326410248b4dc9837bcf9d3d2"
server_key_iv: "5f3b96c4253f0029ceea9ee85a6668a8"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.759037 3172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d4456c0c033845cea16ac2695d5bf0ac"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "1c86957326410248b4dc9837bcf9d3d2"
server_key_iv: "5f3b96c4253f0029ceea9ee85a6668a8"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.763016 3172 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:13.765547 3187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.766789 3172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:13.766925 3172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "d4456c0c033845cea16ac2695d5bf0ac"
format_stamp: "Formatted at 2026-09-28 21:12:13 on dist-test-slave-v7xz"
server_key: "1c86957326410248b4dc9837bcf9d3d2"
server_key_iv: "5f3b96c4253f0029ceea9ee85a6668a8"
server_key_version: "encryptionkey@0"
I20260928 21:12:13.767035 3172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:13.789970 3172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:13.792888 3172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:13.793093 3172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:13.793689 3172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:13.794512 3172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:13.794581 3172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.794655 3172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:13.794699 3172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:13.803198 3172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38353
I20260928 21:12:13.803251 3300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38353 every 8 connection(s)
I20260928 21:12:13.804095 3172 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:13.813232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3172
I20260928 21:12:13.813365 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:13.813647 392 external_mini_cluster.cc:1428] Setting key 36acbf590c6b28629ef6b21d96d3f9f8
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[2797](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629933, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:13.818199 2881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.805923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33813 (local address 127.0.98.62:34367)
0928 21:12:13.806080 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:13.806084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:13.806888 (+ 804us) server_negotiation.cc:407] Connection header received
0928 21:12:13.808281 (+ 1393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:13.808285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:13.808342 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:13.808422 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:13.809702 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.810321 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.811571 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.811796 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.811918 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.814553 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:13.814562 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:13.814564 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:13.814588 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:13.816117 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.816645 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.816650 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.816652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.816734 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:13.817114 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:13.817119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:13.817120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:13.817302 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:13.817417 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:13.817922 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 21:12:13.818041 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:12:13.819032 3303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:13.806292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34367 (local address 127.0.98.3:33813)
0928 21:12:13.806726 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:13.806769 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:13.808057 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:13.808597 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:13.808604 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:13.809044 (+ 440us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:13.809550 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:13.809564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.810480 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.810483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:13.811451 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:13.811460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:13.811981 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:13.811986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:13.812166 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:13.812792 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:13.812807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:13.814383 (+ 1576us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:13.816237 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:13.816243 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:13.816260 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:13.816536 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:13.816857 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:13.816860 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:13.816863 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:13.817014 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:13.817415 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:13.817422 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:13.817674 (+ 252us) client_negotiation.cc:769] Sending connection context
0928 21:12:13.817865 (+ 191us) client_negotiation.cc:240] Negotiation successful
0928 21:12:13.818202 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":119,"threads_started":1}
I20260928 21:12:13.820138 3301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34367
I20260928 21:12:13.820370 3301 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:13.820816 3301 heartbeater.cc:511] Master 127.0.98.62:34367 requested a full tablet report, sending...
I20260928 21:12:13.821972 2842 ts_manager.cc:194] Registered new tserver with Master: d4456c0c033845cea16ac2695d5bf0ac (127.0.98.3:38353)
I20260928 21:12:13.822490 2842 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33813
I20260928 21:12:13.827452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:12:13.840560 2881 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:13.838975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45954 (local address 127.0.98.62:34367)
0928 21:12:13.839178 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:13.839183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:13.839342 (+ 159us) server_negotiation.cc:407] Connection header received
0928 21:12:13.839505 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:13.839508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:13.839600 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:13.839706 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:13.840403 (+ 697us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45954: BlockingRecv error: recv got EOF from 127.0.0.1:45954 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:12:13.841596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 2904
I20260928 21:12:13.846848 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3038
I20260928 21:12:13.851999 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3172
I20260928 21:12:13.857427 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 2810
2026-09-28T21:12:13Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2119 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[3313](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[3313](info): set up 2 sockets
Sep 28 21:12:13 dist-test-slave-v7xz krb5kdc[3313](info): commencing operation
krb5kdc: starting...
W20260928 21:12:14.900717 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:14 dist-test-slave-v7xz krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:15.023821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:15.135954 3326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:15.136260 3326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:15.136302 3326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:15.140417 3326 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:15.140496 3326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:15.140515 3326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:15.140532 3326 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:15.140548 3326 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:15.146235 3326 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:15.147339 3326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:15.148348 3326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:15.156975 3332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:15.157004 3334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:15.157035 3331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:15.157402 3326 server_base.cc:1034] running on GCE node
I20260928 21:12:15.158052 3326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:15.159219 3326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:15.160415 3326 hybrid_clock.cc:648] HybridClock initialized: now 1790629935160383 us; error 50 us; skew 500 ppm
Sep 28 21:12:15 dist-test-slave-v7xz krb5kdc[3313](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:15.163390 3326 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:15.164315 3326 webserver.cc:460] Webserver started at http://127.0.98.62:43771/ using document root <none> and password file <none>
I20260928 21:12:15.164988 3326 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:15.165086 3326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:15.165328 3326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:15.167430 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "8f756530862843c99134645fd69b2137"
format_stamp: "Formatted at 2026-09-28 21:12:15 on dist-test-slave-v7xz"
server_key: "e1bd81e9ac95f4b0cc5a11f455265044"
server_key_iv: "791ce3e88dc2939c56c740387dd99b35"
server_key_version: "encryptionkey@0"
I20260928 21:12:15.168025 3326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "8f756530862843c99134645fd69b2137"
format_stamp: "Formatted at 2026-09-28 21:12:15 on dist-test-slave-v7xz"
server_key: "e1bd81e9ac95f4b0cc5a11f455265044"
server_key_iv: "791ce3e88dc2939c56c740387dd99b35"
server_key_version: "encryptionkey@0"
I20260928 21:12:15.171851 3326 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260928 21:12:15.174340 3341 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:15.175557 3326 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:15.175698 3326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "8f756530862843c99134645fd69b2137"
format_stamp: "Formatted at 2026-09-28 21:12:15 on dist-test-slave-v7xz"
server_key: "e1bd81e9ac95f4b0cc5a11f455265044"
server_key_iv: "791ce3e88dc2939c56c740387dd99b35"
server_key_version: "encryptionkey@0"
I20260928 21:12:15.175812 3326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:15.197389 3326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:15.200927 3326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:15.201138 3326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:15.208207 3326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46049
I20260928 21:12:15.208489 3393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46049 every 8 connection(s)
I20260928 21:12:15.209252 3326 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:15.210515 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3326
I20260928 21:12:15.210666 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:15.210991 392 external_mini_cluster.cc:1428] Setting key cb97abc386bfde9ae6703bde7f0c7a6e
I20260928 21:12:15.212643 3394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:15.218088 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137: Bootstrap starting.
Sep 28 21:12:15 dist-test-slave-v7xz krb5kdc[3313](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:15.220799 3394 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:15.221692 3394 log.cc:824] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:15.223539 3394 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137: No bootstrap required, opened a new log
I20260928 21:12:15.225244 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:15.212296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.98.62:46049)
0928 21:12:15.212803 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:15.212820 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:15.212943 (+ 123us) server_negotiation.cc:407] Connection header received
0928 21:12:15.213588 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:15.213605 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:15.213937 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:15.214278 (+ 341us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:15.215103 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:15.216092 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:15.217008 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:15.217512 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:15.217723 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:15.220464 (+ 2741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:15.220469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:15.220483 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:15.220510 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:15.222282 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:15.222752 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:15.222757 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:15.222765 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:15.222880 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:15.223165 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:15.223170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:15.223171 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:15.223577 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:15.223785 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:15.224124 (+ 339us) server_negotiation.cc:299] Negotiation successful
0928 21:12:15.224458 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":141,"threads_started":1}
I20260928 21:12:15.227236 3394 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f756530862843c99134645fd69b2137" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260928 21:12:15.227541 3394 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:15.227640 3394 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f756530862843c99134645fd69b2137, State: Initialized, Role: FOLLOWER
I20260928 21:12:15.228050 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:12:15.228087 3394 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f756530862843c99134645fd69b2137" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260928 21:12:15.228278 3394 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:15.228343 3394 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:15.228438 3394 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:15.229416 3394 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f756530862843c99134645fd69b2137" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260928 21:12:15.229753 3394 leader_election.cc:302] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f756530862843c99134645fd69b2137; no voters:
I20260928 21:12:15.230083 3394 leader_election.cc:288] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:15.230154 3400 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:15.230396 3400 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [term 1 LEADER]: Becoming Leader. State: Replica: 8f756530862843c99134645fd69b2137, State: Running, Role: LEADER
I20260928 21:12:15.230742 3400 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f756530862843c99134645fd69b2137" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } }
I20260928 21:12:15.231348 3394 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:15.231889 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f756530862843c99134645fd69b2137. Latest consensus state: current_term: 1 leader_uuid: "8f756530862843c99134645fd69b2137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f756530862843c99134645fd69b2137" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } } }
I20260928 21:12:15.232653 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:15.232523 3401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f756530862843c99134645fd69b2137" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f756530862843c99134645fd69b2137" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46049 } } }
I20260928 21:12:15.232998 3401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:15.233760 3409 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:15.237412 3409 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:15.243680 3409 catalog_manager.cc:1347] Generated new cluster ID: cb76a854aef24518b67fdcef85e74314
I20260928 21:12:15.243744 3409 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:15.246573 3397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:15.241362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43690 (local address 127.0.98.62:46049)
0928 21:12:15.241490 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:15.241495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:15.241783 (+ 288us) server_negotiation.cc:407] Connection header received
0928 21:12:15.241922 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:15.241925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:15.241978 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:15.242076 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:15.242780 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:15.243523 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:15.244440 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:15.244678 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:15.244795 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:15.245720 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:15.245724 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:15.245726 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:15.246131 (+ 405us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:15.246159 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:15.246175 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:15.246277 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:15.246359 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:12:15.246425 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:12:15.262312 3409 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:15.263571 3409 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:15.272014 3409 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f756530862843c99134645fd69b2137: Generated new TSK 0
I20260928 21:12:15.272804 3409 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:15.282351 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3326
2026-09-28T21:12:15Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1422 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:15 dist-test-slave-v7xz krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:15 dist-test-slave-v7xz krb5kdc[3427](info): set up 2 sockets
Sep 28 21:12:15 dist-test-slave-v7xz krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20260928 21:12:16.313095 392 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:16 dist-test-slave-v7xz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:16.439850 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34877
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34877
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:16.551661 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:16.551949 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:16.551990 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:16.556023 3440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:16.556094 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:16.556114 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:16.556131 3440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:16.556147 3440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:16.561424 3440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34877
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:16.562423 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:16.563367 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:16.571652 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:16.571668 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:16.571655 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:16.572082 3440 server_base.cc:1034] running on GCE node
I20260928 21:12:16.572778 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:16.574282 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:16.575490 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790629936575470 us; error 50 us; skew 500 ppm
Sep 28 21:12:16 dist-test-slave-v7xz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:16.579205 3440 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:16.580356 3440 webserver.cc:460] Webserver started at http://127.0.98.62:42019/ using document root <none> and password file <none>
I20260928 21:12:16.581125 3440 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:16.581210 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:16.581440 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:16.584075 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "d621a4277af745c19387f9763fd984fd"
format_stamp: "Formatted at 2026-09-28 21:12:16 on dist-test-slave-v7xz"
server_key: "fe400ea7e04174cd6c2a02e6f79e9c87"
server_key_iv: "7035d50fdbf04859902b6bccdd9cc5ac"
server_key_version: "encryptionkey@0"
I20260928 21:12:16.584589 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "d621a4277af745c19387f9763fd984fd"
format_stamp: "Formatted at 2026-09-28 21:12:16 on dist-test-slave-v7xz"
server_key: "fe400ea7e04174cd6c2a02e6f79e9c87"
server_key_iv: "7035d50fdbf04859902b6bccdd9cc5ac"
server_key_version: "encryptionkey@0"
I20260928 21:12:16.588297 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:12:16.590744 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:16.591967 3440 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:16.592099 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "d621a4277af745c19387f9763fd984fd"
format_stamp: "Formatted at 2026-09-28 21:12:16 on dist-test-slave-v7xz"
server_key: "fe400ea7e04174cd6c2a02e6f79e9c87"
server_key_iv: "7035d50fdbf04859902b6bccdd9cc5ac"
server_key_version: "encryptionkey@0"
I20260928 21:12:16.592204 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:16.608048 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:16.611542 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:16.611756 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:16.618561 3440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34877
I20260928 21:12:16.618829 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34877 every 8 connection(s)
I20260928 21:12:16.619563 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:16.622351 3508 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:16.626398 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3440
I20260928 21:12:16.626555 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:16.626845 392 external_mini_cluster.cc:1428] Setting key d46a248dca6b5ee7460028ccddb4b6ad
I20260928 21:12:16.628268 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd: Bootstrap starting.
I20260928 21:12:16.630785 3508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:16.631544 3508 log.cc:824] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:16.633531 3508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd: No bootstrap required, opened a new log
Sep 28 21:12:16 dist-test-slave-v7xz krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629936, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:16.637292 3508 raft_consensus.cc:348] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621a4277af745c19387f9763fd984fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34877 } }
I20260928 21:12:16.637607 3508 raft_consensus.cc:374] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:16.637691 3508 raft_consensus.cc:729] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d621a4277af745c19387f9763fd984fd, State: Initialized, Role: FOLLOWER
I20260928 21:12:16.638190 3508 consensus_queue.cc:260] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621a4277af745c19387f9763fd984fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34877 } }
I20260928 21:12:16.638351 3508 raft_consensus.cc:388] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:16.638428 3508 raft_consensus.cc:482] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:16.638552 3508 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:16.639482 3508 raft_consensus.cc:504] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621a4277af745c19387f9763fd984fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34877 } }
I20260928 21:12:16.639834 3508 leader_election.cc:302] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d621a4277af745c19387f9763fd984fd; no voters:
I20260928 21:12:16.640198 3508 leader_election.cc:288] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:16.640328 3513 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:16.640558 3513 raft_consensus.cc:686] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [term 1 LEADER]: Becoming Leader. State: Replica: d621a4277af745c19387f9763fd984fd, State: Running, Role: LEADER
I20260928 21:12:16.640931 3513 consensus_queue.cc:237] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621a4277af745c19387f9763fd984fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34877 } }
I20260928 21:12:16.641562 3508 sys_catalog.cc:564] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:16.641563 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:16.628198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45930 (local address 127.0.98.62:34877)
0928 21:12:16.628641 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:16.628659 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:16.628705 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:12:16.629412 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:16.629429 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:16.629764 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:16.630053 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:16.631019 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:16.632072 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:16.633042 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:16.633525 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:16.633697 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:16.636229 (+ 2532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:16.636235 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:16.636251 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:16.636280 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:16.638228 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:16.638784 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:16.638789 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:16.638794 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:16.638910 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:16.639216 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:16.639220 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:16.639221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:16.639689 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:16.639906 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:16.640177 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 21:12:16.640532 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":176,"threads_started":1}
I20260928 21:12:16.642145 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d621a4277af745c19387f9763fd984fd. Latest consensus state: current_term: 1 leader_uuid: "d621a4277af745c19387f9763fd984fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621a4277af745c19387f9763fd984fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34877 } } }
I20260928 21:12:16.642263 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:16.642655 3518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:16.644789 3515 sys_catalog.cc:455] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d621a4277af745c19387f9763fd984fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d621a4277af745c19387f9763fd984fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34877 } } }
I20260928 21:12:16.644935 3515 sys_catalog.cc:458] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:16.646118 3518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:16.652173 3518 catalog_manager.cc:1347] Generated new cluster ID: 56c5c109536f4d9d87b1519613f59d7c
I20260928 21:12:16.652268 3518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:16.666167 3518 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:16.667519 3518 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:16.674546 3518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d621a4277af745c19387f9763fd984fd: Generated new TSK 0
I20260928 21:12:16.675420 3518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:16.706132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34877
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:16.819058 3534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:16.819365 3534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:16.819455 3534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:16.823547 3534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:16.823643 3534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:16.823737 3534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:16.829160 3534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:16.830251 3534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:16.831334 3534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:16.840864 3539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:16.840870 3540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:16.840879 3542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:16.841259 3534 server_base.cc:1034] running on GCE node
I20260928 21:12:16.841681 3534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:16.842437 3534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:16.843623 3534 hybrid_clock.cc:648] HybridClock initialized: now 1790629936843601 us; error 39 us; skew 500 ppm
Sep 28 21:12:16 dist-test-slave-v7xz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:16.849664 3534 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:16.850564 3534 webserver.cc:460] Webserver started at http://127.0.98.1:38955/ using document root <none> and password file <none>
I20260928 21:12:16.851135 3534 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:16.851226 3534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:16.851459 3534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:16.853544 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "bdcc23d82a834e4abc583aaaf262b42b"
format_stamp: "Formatted at 2026-09-28 21:12:16 on dist-test-slave-v7xz"
server_key: "beea2d8182aa2ef1cdd98291af436d6f"
server_key_iv: "8c77d8502ffe778181f9581e2ba02169"
server_key_version: "encryptionkey@0"
I20260928 21:12:16.854110 3534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bdcc23d82a834e4abc583aaaf262b42b"
format_stamp: "Formatted at 2026-09-28 21:12:16 on dist-test-slave-v7xz"
server_key: "beea2d8182aa2ef1cdd98291af436d6f"
server_key_iv: "8c77d8502ffe778181f9581e2ba02169"
server_key_version: "encryptionkey@0"
I20260928 21:12:16.857918 3534 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:16.860441 3549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:16.861706 3534 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:16.861845 3534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "bdcc23d82a834e4abc583aaaf262b42b"
format_stamp: "Formatted at 2026-09-28 21:12:16 on dist-test-slave-v7xz"
server_key: "beea2d8182aa2ef1cdd98291af436d6f"
server_key_iv: "8c77d8502ffe778181f9581e2ba02169"
server_key_version: "encryptionkey@0"
I20260928 21:12:16.861976 3534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:16.874078 3534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:16.877559 3534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:16.877774 3534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:16.878408 3534 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:16.879248 3534 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:16.879329 3534 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:16.879401 3534 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:16.879436 3534 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:16.888931 3534 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41619
I20260928 21:12:16.888948 3662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41619 every 8 connection(s)
I20260928 21:12:16.889876 3534 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:16.892933 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3534
I20260928 21:12:16.893062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:16.893393 392 external_mini_cluster.cc:1428] Setting key 94c007aba88004dbe7f3a8bb85694745
Sep 28 21:12:16 dist-test-slave-v7xz krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629936, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:16.905982 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:16.891752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34485 (local address 127.0.98.62:34877)
0928 21:12:16.891895 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:16.891898 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:16.892884 (+ 986us) server_negotiation.cc:407] Connection header received
0928 21:12:16.894657 (+ 1773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:16.894660 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:16.894706 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:16.894799 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:16.896540 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:16.897149 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:16.898998 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:16.899216 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:16.899330 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:16.901979 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:16.901985 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:16.901992 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:16.902018 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:16.903624 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:16.904142 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:16.904144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:16.904146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:16.904224 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:16.904571 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:16.904574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:16.904575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:16.904773 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:16.905017 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:16.905680 (+ 663us) server_negotiation.cc:299] Negotiation successful
0928 21:12:16.905836 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:12:16.906965 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:16.892167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34877 (local address 127.0.98.1:34485)
0928 21:12:16.892666 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:16.892731 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:16.894439 (+ 1708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:16.895027 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:16.895036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:16.895605 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:16.896376 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:16.896399 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:16.897291 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:16.897297 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:16.898865 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:16.898880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:16.899392 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:16.899396 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:16.899579 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:16.900221 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:16.900239 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:16.901784 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:16.903734 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:16.903738 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:16.903752 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:16.904041 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:16.904310 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:16.904312 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:16.904313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:16.904457 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:16.905011 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:16.905016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:16.905264 (+ 248us) client_negotiation.cc:769] Sending connection context
0928 21:12:16.905431 (+ 167us) client_negotiation.cc:240] Negotiation successful
0928 21:12:16.905801 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":116,"threads_started":1}
I20260928 21:12:16.908385 3663 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34877
I20260928 21:12:16.908761 3663 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:16.909442 3663 heartbeater.cc:511] Master 127.0.98.62:34877 requested a full tablet report, sending...
I20260928 21:12:16.911386 3472 ts_manager.cc:194] Registered new tserver with Master: bdcc23d82a834e4abc583aaaf262b42b (127.0.98.1:41619)
I20260928 21:12:16.913188 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34485
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:16.924216 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34877
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:17.043306 3668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:17.043644 3668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:17.043742 3668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:17.048146 3668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:17.048247 3668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:17.048339 3668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:17.057459 3668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:17.058743 3668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:17.059736 3668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:17.069852 3668 server_base.cc:1034] running on GCE node
W20260928 21:12:17.069806 3676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:17.069814 3674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:17.069834 3673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:17.070479 3668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:17.071185 3668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:17.072381 3668 hybrid_clock.cc:648] HybridClock initialized: now 1790629937072360 us; error 34 us; skew 500 ppm
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:17.075439 3668 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:17.076463 3668 webserver.cc:460] Webserver started at http://127.0.98.2:33001/ using document root <none> and password file <none>
I20260928 21:12:17.077127 3668 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:17.077221 3668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:17.077456 3668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:17.079605 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "249797f83bf549378278ab23643cbb27"
format_stamp: "Formatted at 2026-09-28 21:12:17 on dist-test-slave-v7xz"
server_key: "b1e0f72cb02a19a22466d3a0b43df521"
server_key_iv: "4ba015124a5b3ff46d36ebccb4a6e34f"
server_key_version: "encryptionkey@0"
I20260928 21:12:17.080212 3668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "249797f83bf549378278ab23643cbb27"
format_stamp: "Formatted at 2026-09-28 21:12:17 on dist-test-slave-v7xz"
server_key: "b1e0f72cb02a19a22466d3a0b43df521"
server_key_iv: "4ba015124a5b3ff46d36ebccb4a6e34f"
server_key_version: "encryptionkey@0"
I20260928 21:12:17.084689 3668 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:17.087492 3683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:17.088971 3668 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:17.089115 3668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "249797f83bf549378278ab23643cbb27"
format_stamp: "Formatted at 2026-09-28 21:12:17 on dist-test-slave-v7xz"
server_key: "b1e0f72cb02a19a22466d3a0b43df521"
server_key_iv: "4ba015124a5b3ff46d36ebccb4a6e34f"
server_key_version: "encryptionkey@0"
I20260928 21:12:17.089242 3668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:17.108141 3668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:17.111756 3668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:17.111981 3668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:17.112601 3668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:17.113517 3668 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:17.113588 3668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:17.113662 3668 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:17.113705 3668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:17.123463 3668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39979
I20260928 21:12:17.123485 3796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39979 every 8 connection(s)
I20260928 21:12:17.124480 3668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:17.131378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3668
I20260928 21:12:17.131520 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:17.131871 392 external_mini_cluster.cc:1428] Setting key 9bcadd069a0033880e4cf98a9e17df0b
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:17.140111 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:17.126566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55231 (local address 127.0.98.62:34877)
0928 21:12:17.126713 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:17.126716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:17.127619 (+ 903us) server_negotiation.cc:407] Connection header received
0928 21:12:17.129061 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:17.129064 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:17.129113 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:17.129190 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:17.130408 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.130939 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.132664 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.133003 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.133112 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:17.136113 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:17.136119 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:17.136124 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:17.136158 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:17.137934 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:17.138511 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:17.138514 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:17.138518 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:17.138603 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:17.138958 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:17.138963 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:17.138965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:17.139159 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:17.139270 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:17.139801 (+ 531us) server_negotiation.cc:299] Negotiation successful
0928 21:12:17.139939 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":56}
I20260928 21:12:17.141233 3799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:17.126976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34877 (local address 127.0.98.2:55231)
0928 21:12:17.127454 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:17.127498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:17.128793 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:17.129400 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:17.129408 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:17.129771 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:17.130263 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:17.130276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.131096 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.131100 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:17.132521 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:17.132532 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.133252 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.133258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:17.133495 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:17.134137 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:17.134163 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:17.135925 (+ 1762us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:17.138065 (+ 2140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:17.138071 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:17.138090 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:17.138366 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:17.138699 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:17.138703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:17.138705 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:17.138857 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:17.139269 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:17.139278 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:17.139558 (+ 280us) client_negotiation.cc:769] Sending connection context
0928 21:12:17.139762 (+ 204us) client_negotiation.cc:240] Negotiation successful
0928 21:12:17.140122 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:17.142516 3797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34877
I20260928 21:12:17.142789 3797 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:17.143386 3797 heartbeater.cc:511] Master 127.0.98.62:34877 requested a full tablet report, sending...
I20260928 21:12:17.144443 3472 ts_manager.cc:194] Registered new tserver with Master: 249797f83bf549378278ab23643cbb27 (127.0.98.2:39979)
I20260928 21:12:17.145103 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
W20260928 21:12:17.464552 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.3: real 0.331s user 0.000s sys 0.004s
I20260928 21:12:17.465096 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34877
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:17.576330 3802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:17.576692 3802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:17.576781 3802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:17.580909 3802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:17.581024 3802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:17.581127 3802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:17.586639 3802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34877
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:17.587818 3802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:17.588898 3802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:17.598774 3810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:17.598814 3807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:17.598814 3808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:17.599005 3802 server_base.cc:1034] running on GCE node
I20260928 21:12:17.599427 3802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:17.600148 3802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:17.601356 3802 hybrid_clock.cc:648] HybridClock initialized: now 1790629937601345 us; error 35 us; skew 500 ppm
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:17.604240 3802 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:17.605139 3802 webserver.cc:460] Webserver started at http://127.0.98.3:41769/ using document root <none> and password file <none>
I20260928 21:12:17.605692 3802 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:17.605753 3802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:17.605921 3802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:17.607923 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "1a57eeffe544481ebb1a45b3fdaee2c3"
format_stamp: "Formatted at 2026-09-28 21:12:17 on dist-test-slave-v7xz"
server_key: "6d8b4946737ff425abe29257071ab105"
server_key_iv: "9abae26bb26a6eb9efec912994269961"
server_key_version: "encryptionkey@0"
I20260928 21:12:17.608444 3802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1a57eeffe544481ebb1a45b3fdaee2c3"
format_stamp: "Formatted at 2026-09-28 21:12:17 on dist-test-slave-v7xz"
server_key: "6d8b4946737ff425abe29257071ab105"
server_key_iv: "9abae26bb26a6eb9efec912994269961"
server_key_version: "encryptionkey@0"
I20260928 21:12:17.612308 3802 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:17.614890 3817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:17.616214 3802 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:17.616321 3802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "1a57eeffe544481ebb1a45b3fdaee2c3"
format_stamp: "Formatted at 2026-09-28 21:12:17 on dist-test-slave-v7xz"
server_key: "6d8b4946737ff425abe29257071ab105"
server_key_iv: "9abae26bb26a6eb9efec912994269961"
server_key_version: "encryptionkey@0"
I20260928 21:12:17.616492 3802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:17.646263 3802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:17.650004 3802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:17.650226 3802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:17.650856 3802 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:17.651793 3802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:17.651840 3802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:17.651909 3802 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:17.651935 3802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:17.660933 3802 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37671
I20260928 21:12:17.660977 3930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37671 every 8 connection(s)
I20260928 21:12:17.661883 3802 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:17.662248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3802
I20260928 21:12:17.662381 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:17.662688 392 external_mini_cluster.cc:1428] Setting key 47a1636c5955de0f81c8b87d2d309b2f
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629937, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:17.676263 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:17.663801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49161 (local address 127.0.98.62:34877)
0928 21:12:17.664053 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:17.664057 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:17.665223 (+ 1166us) server_negotiation.cc:407] Connection header received
0928 21:12:17.666715 (+ 1492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:17.666718 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:17.666770 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:17.666853 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:17.668055 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.668586 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.669808 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.670044 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.670126 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:17.672515 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:17.672518 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:17.672520 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:17.672544 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:17.674122 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:17.674559 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:17.674561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:17.674562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:17.674652 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:17.675061 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:17.675069 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:17.675070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:17.675249 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:17.675359 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:17.675921 (+ 562us) server_negotiation.cc:299] Negotiation successful
0928 21:12:17.676079 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260928 21:12:17.677192 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:17.664405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34877 (local address 127.0.98.3:49161)
0928 21:12:17.665061 (+ 656us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:17.665125 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:17.666459 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:17.666999 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:17.667007 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:17.667386 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:17.667915 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:17.667928 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.668696 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.668698 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:17.669696 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:17.669705 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.670177 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.670179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:17.670327 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:17.670872 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:17.670888 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:17.672357 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:17.674217 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:17.674221 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:17.674233 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:17.674476 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:17.674787 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:17.674790 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:17.674792 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:17.674948 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:17.675357 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:17.675364 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:17.675628 (+ 264us) client_negotiation.cc:769] Sending connection context
0928 21:12:17.675844 (+ 216us) client_negotiation.cc:240] Negotiation successful
0928 21:12:17.676187 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":157,"threads_started":1}
I20260928 21:12:17.678236 3931 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34877
I20260928 21:12:17.678493 3931 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:17.679003 3931 heartbeater.cc:511] Master 127.0.98.62:34877 requested a full tablet report, sending...
I20260928 21:12:17.679957 3472 ts_manager.cc:194] Registered new tserver with Master: 1a57eeffe544481ebb1a45b3fdaee2c3 (127.0.98.3:37671)
I20260928 21:12:17.680519 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49161
I20260928 21:12:17.688690 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629937, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629937, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:17.714458 3933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:17.705635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45934 (local address 127.0.98.62:34877)
0928 21:12:17.705816 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:17.705821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:17.705961 (+ 140us) server_negotiation.cc:407] Connection header received
0928 21:12:17.706138 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:17.706141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:17.706186 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:17.706277 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:17.707004 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.707536 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.708378 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:17.708613 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:17.708731 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:17.711193 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:17.711196 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:17.711198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:17.711225 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:17.712710 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:17.713171 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:17.713174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:17.713176 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:17.713256 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:17.713540 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:17.713544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:17.713546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:17.713777 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:17.713931 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:17.714159 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 21:12:17.714288 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
W20260928 21:12:17.714910 3472 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:45934
I20260928 21:12:17.715906 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3534
I20260928 21:12:17.721625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3668
I20260928 21:12:17.726934 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3802
I20260928 21:12:17.732259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3440
2026-09-28T21:12:17Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2452 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3944](info): set up 2 sockets
Sep 28 21:12:17 dist-test-slave-v7xz krb5kdc[3944](info): commencing operation
krb5kdc: starting...
W20260928 21:12:18.828990 392 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:18 dist-test-slave-v7xz krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:18.952293 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36553
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:19.064977 3957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:19.065289 3957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:19.065333 3957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:19.069521 3957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:19.069591 3957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:19.069612 3957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:19.069628 3957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:19.069644 3957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:19.075169 3957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:19.076305 3957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:19.077409 3957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:19.086048 3965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:19.086124 3962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:19.086071 3963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:19.086261 3957 server_base.cc:1034] running on GCE node
I20260928 21:12:19.086951 3957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:19.088200 3957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:19.089383 3957 hybrid_clock.cc:648] HybridClock initialized: now 1790629939089369 us; error 27 us; skew 500 ppm
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:19.092298 3957 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:19.093326 3957 webserver.cc:460] Webserver started at http://127.0.98.62:35173/ using document root <none> and password file <none>
I20260928 21:12:19.093930 3957 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:19.094025 3957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:19.094269 3957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:19.096434 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "e1bfa991572d4f80963816ff110bc599"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "8fbd14c6c121598f6936f7e5a8500f13"
server_key_iv: "c06214659719b6242efba55cfa8df26f"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.097067 3957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "e1bfa991572d4f80963816ff110bc599"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "8fbd14c6c121598f6936f7e5a8500f13"
server_key_iv: "c06214659719b6242efba55cfa8df26f"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.101038 3957 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:12:19.103513 3972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.104776 3957 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:19.104964 3957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "e1bfa991572d4f80963816ff110bc599"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "8fbd14c6c121598f6936f7e5a8500f13"
server_key_iv: "c06214659719b6242efba55cfa8df26f"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.105090 3957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:19.136180 3957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:19.139660 3957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:19.139876 3957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:19.147181 3957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36553
I20260928 21:12:19.147209 4024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36553 every 8 connection(s)
I20260928 21:12:19.148186 3957 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:19.149338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 3957
I20260928 21:12:19.149478 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:19.149776 392 external_mini_cluster.cc:1428] Setting key a5973eeceb0b73a5431cddcf827a2539
I20260928 21:12:19.151193 4025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:19.157774 4025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599: Bootstrap starting.
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:19.160588 4025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:19.161396 4025 log.cc:824] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:19.163365 4025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599: No bootstrap required, opened a new log
I20260928 21:12:19.165841 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.151168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38998 (local address 127.0.98.62:36553)
0928 21:12:19.151656 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:19.151673 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:19.151872 (+ 199us) server_negotiation.cc:407] Connection header received
0928 21:12:19.152668 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:19.152685 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:19.153238 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:19.153609 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:19.154811 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.155779 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.157003 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.157540 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.157679 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.160550 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:19.160564 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:19.160578 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:19.160606 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:19.162640 (+ 2034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.163085 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.163091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.163097 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.163216 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.163536 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.163540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.163541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.164046 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:19.164236 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:19.164488 (+ 252us) server_negotiation.cc:299] Negotiation successful
0928 21:12:19.164780 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":149,"threads_started":1}
I20260928 21:12:19.167804 4025 raft_consensus.cc:348] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bfa991572d4f80963816ff110bc599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260928 21:12:19.168109 4025 raft_consensus.cc:374] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:19.168171 4025 raft_consensus.cc:729] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1bfa991572d4f80963816ff110bc599, State: Initialized, Role: FOLLOWER
I20260928 21:12:19.168650 4025 consensus_queue.cc:260] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bfa991572d4f80963816ff110bc599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260928 21:12:19.168802 4025 raft_consensus.cc:388] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:19.168905 4025 raft_consensus.cc:482] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:19.169023 4025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:19.169924 4025 raft_consensus.cc:504] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bfa991572d4f80963816ff110bc599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260928 21:12:19.170253 4025 leader_election.cc:302] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e1bfa991572d4f80963816ff110bc599; no voters:
I20260928 21:12:19.170553 4025 leader_election.cc:288] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:19.170691 4030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:19.170948 4030 raft_consensus.cc:686] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [term 1 LEADER]: Becoming Leader. State: Replica: e1bfa991572d4f80963816ff110bc599, State: Running, Role: LEADER
I20260928 21:12:19.171278 4030 consensus_queue.cc:237] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bfa991572d4f80963816ff110bc599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } }
I20260928 21:12:19.171737 4025 sys_catalog.cc:564] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:19.172940 4032 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e1bfa991572d4f80963816ff110bc599. Latest consensus state: current_term: 1 leader_uuid: "e1bfa991572d4f80963816ff110bc599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bfa991572d4f80963816ff110bc599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } } }
I20260928 21:12:19.173080 4032 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:19.173384 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e1bfa991572d4f80963816ff110bc599" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e1bfa991572d4f80963816ff110bc599" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36553 } } }
I20260928 21:12:19.173506 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:19.173769 4039 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:19.176638 4039 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:19.181999 4039 catalog_manager.cc:1347] Generated new cluster ID: 24be3c98e8354354b50c7756118674b9
I20260928 21:12:19.182070 4039 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:19.209398 4039 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:19.210534 4039 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:19.220029 4039 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e1bfa991572d4f80963816ff110bc599: Generated new TSK 0
I20260928 21:12:19.220901 4039 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:19.252051 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36553
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:19.371565 4051 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:19.371847 4051 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:19.371891 4051 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:19.376135 4051 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:19.376209 4051 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:19.376269 4051 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:19.381609 4051 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4051
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:19.382658 4051 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:19.383648 4051 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:19.393219 4056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:19.393276 4057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:19.393330 4051 server_base.cc:1034] running on GCE node
W20260928 21:12:19.393261 4059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:19.393780 4051 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:19.394443 4051 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:19.395619 4051 hybrid_clock.cc:648] HybridClock initialized: now 1790629939395606 us; error 28 us; skew 500 ppm
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:19.398550 4051 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:19.399439 4051 webserver.cc:460] Webserver started at http://127.0.98.1:37595/ using document root <none> and password file <none>
I20260928 21:12:19.399958 4051 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:19.400018 4051 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:19.400180 4051 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:19.402518 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "1fbc4b0532fb4ca0a9b0a8ca70417aca"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "ffa8ff5de25cdede16cdfb343d5d6fd2"
server_key_iv: "b78cf67f7102e349b5c968942620e003"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.403123 4051 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1fbc4b0532fb4ca0a9b0a8ca70417aca"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "ffa8ff5de25cdede16cdfb343d5d6fd2"
server_key_iv: "b78cf67f7102e349b5c968942620e003"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.407147 4051 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:19.409936 4066 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.411190 4051 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:19.411321 4051 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "1fbc4b0532fb4ca0a9b0a8ca70417aca"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "ffa8ff5de25cdede16cdfb343d5d6fd2"
server_key_iv: "b78cf67f7102e349b5c968942620e003"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.411442 4051 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:19.452663 4051 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:19.456203 4051 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:19.456418 4051 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:19.457056 4051 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:19.457918 4051 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:19.457991 4051 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.458063 4051 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:19.458104 4051 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.467051 4051 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37831
I20260928 21:12:19.467077 4179 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37831 every 8 connection(s)
I20260928 21:12:19.468070 4051 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:19.469403 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4051
I20260928 21:12:19.469529 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:19.469830 392 external_mini_cluster.cc:1428] Setting key d582d577c876f4f43ce7d11e177745f8
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:19.484560 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.470182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38599 (local address 127.0.98.62:36553)
0928 21:12:19.470342 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:19.470347 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:19.471356 (+ 1009us) server_negotiation.cc:407] Connection header received
0928 21:12:19.473529 (+ 2173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:19.473535 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:19.473594 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:19.473697 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:19.475178 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.475749 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.477684 (+ 1935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.477910 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.477995 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.480779 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:19.480783 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:19.480786 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:19.480818 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:19.482330 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.482947 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.482951 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.482953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.483063 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.483427 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.483430 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.483432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.483625 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:19.483730 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:19.484252 (+ 522us) server_negotiation.cc:299] Negotiation successful
0928 21:12:19.484386 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:12:19.485471 4182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.470628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36553 (local address 127.0.98.1:38599)
0928 21:12:19.471203 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:19.471264 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:19.473272 (+ 2008us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:19.473886 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:19.473894 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:19.474293 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:19.475015 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:19.475037 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.475898 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.475904 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:19.477554 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:19.477568 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.478081 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.478087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:19.478308 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.478980 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:19.479005 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:19.480603 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:19.482472 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:19.482478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:19.482497 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:19.482827 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:19.483170 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:19.483172 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:19.483174 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:19.483326 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:19.483730 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:19.483738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:19.484019 (+ 281us) client_negotiation.cc:769] Sending connection context
0928 21:12:19.484217 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 21:12:19.484562 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":140,"threads_started":1}
I20260928 21:12:19.486867 4180 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36553
I20260928 21:12:19.487115 4180 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:19.487624 4180 heartbeater.cc:511] Master 127.0.98.62:36553 requested a full tablet report, sending...
I20260928 21:12:19.489240 3989 ts_manager.cc:194] Registered new tserver with Master: 1fbc4b0532fb4ca0a9b0a8ca70417aca (127.0.98.1:37831)
I20260928 21:12:19.490345 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38599
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:19.497965 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36553
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:19.613481 4185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:19.613752 4185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:19.613837 4185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:19.618144 4185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:19.618216 4185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:19.618319 4185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:19.623724 4185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:19.624795 4185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:19.625859 4185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:19.635908 4193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:19.635973 4191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:19.635993 4190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:19.636260 4185 server_base.cc:1034] running on GCE node
I20260928 21:12:19.636894 4185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:19.637589 4185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:19.638777 4185 hybrid_clock.cc:648] HybridClock initialized: now 1790629939638748 us; error 39 us; skew 500 ppm
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:19.646360 4185 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:19.647403 4185 webserver.cc:460] Webserver started at http://127.0.98.2:38263/ using document root <none> and password file <none>
I20260928 21:12:19.648007 4185 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:19.648101 4185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:19.648347 4185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:19.650914 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "21c82d18bbdd42149d59ebd9cf504ab6"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "5042093c27ae6ba16b2eef991ab3451e"
server_key_iv: "ca0c40940aaa3d03c66cfeb22f8f7357"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.651556 4185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "21c82d18bbdd42149d59ebd9cf504ab6"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "5042093c27ae6ba16b2eef991ab3451e"
server_key_iv: "ca0c40940aaa3d03c66cfeb22f8f7357"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.657641 4185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260928 21:12:19.661464 4200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.663173 4185 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:12:19.663321 4185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "21c82d18bbdd42149d59ebd9cf504ab6"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "5042093c27ae6ba16b2eef991ab3451e"
server_key_iv: "ca0c40940aaa3d03c66cfeb22f8f7357"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.663467 4185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:19.696871 4185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:19.700176 4185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:19.700336 4185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:19.700947 4185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:19.701764 4185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:19.701809 4185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.701857 4185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:19.701879 4185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.710516 4185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36463
I20260928 21:12:19.710598 4313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36463 every 8 connection(s)
I20260928 21:12:19.711462 4185 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:19.714951 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4185
I20260928 21:12:19.715035 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:19.715301 392 external_mini_cluster.cc:1428] Setting key 7a6823160d84418b4104c5b330996f34
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:19.726327 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.713392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44087 (local address 127.0.98.62:36553)
0928 21:12:19.713524 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:19.713528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:19.714375 (+ 847us) server_negotiation.cc:407] Connection header received
0928 21:12:19.716078 (+ 1703us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:19.716081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:19.716138 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:19.716223 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:19.717609 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.718204 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.719520 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.719981 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.720105 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.722652 (+ 2547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:19.722655 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:19.722657 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:19.722680 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:19.724131 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.724678 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.724682 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.724685 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.724785 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.725185 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.725188 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.725190 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.725389 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:19.725486 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:19.726100 (+ 614us) server_negotiation.cc:299] Negotiation successful
0928 21:12:19.726207 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:19.727321 4316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.713784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36553 (local address 127.0.98.2:44087)
0928 21:12:19.714228 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:19.714271 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:19.715831 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:19.716415 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:19.716427 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:19.716960 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:19.717466 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:19.717480 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.718352 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.718358 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:19.719406 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:19.719416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.720185 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.720193 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:19.720368 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.721020 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:19.721037 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:19.722474 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:19.724268 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:19.724274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:19.724289 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:19.724548 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:19.724938 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:19.724940 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:19.724942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:19.725074 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:19.725491 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:19.725496 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:19.725827 (+ 331us) client_negotiation.cc:769] Sending connection context
0928 21:12:19.726024 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:12:19.726292 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":112,"threads_started":1}
I20260928 21:12:19.728577 4314 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36553
I20260928 21:12:19.728874 4314 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:19.729396 4314 heartbeater.cc:511] Master 127.0.98.62:36553 requested a full tablet report, sending...
I20260928 21:12:19.730507 3989 ts_manager.cc:194] Registered new tserver with Master: 21c82d18bbdd42149d59ebd9cf504ab6 (127.0.98.2:36463)
I20260928 21:12:19.731091 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:19.743312 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36553
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:19.854012 4319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:19.854270 4319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:19.854350 4319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:19.858374 4319 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:19.858469 4319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:19.858560 4319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:19.863816 4319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36553
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4319
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:19.864962 4319 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:19.866017 4319 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:19.875669 4327 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:19.875689 4324 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:19.876036 4319 server_base.cc:1034] running on GCE node
W20260928 21:12:19.876159 4325 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:19.876603 4319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:19.877357 4319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:19.878537 4319 hybrid_clock.cc:648] HybridClock initialized: now 1790629939878521 us; error 30 us; skew 500 ppm
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:19.881192 4319 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:19.882048 4319 webserver.cc:460] Webserver started at http://127.0.98.3:41605/ using document root <none> and password file <none>
I20260928 21:12:19.882601 4319 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:19.882688 4319 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:19.882911 4319 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:19.884960 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "13a2c5026a774446b7e3696c09d489a3"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "0867720a79f8bfa9f8934a33ca202fb2"
server_key_iv: "c146cbe38ec4f7190574510c4704fa79"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.885520 4319 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "13a2c5026a774446b7e3696c09d489a3"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "0867720a79f8bfa9f8934a33ca202fb2"
server_key_iv: "c146cbe38ec4f7190574510c4704fa79"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.889079 4319 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:19.891522 4334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.892712 4319 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:19.892872 4319 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "13a2c5026a774446b7e3696c09d489a3"
format_stamp: "Formatted at 2026-09-28 21:12:19 on dist-test-slave-v7xz"
server_key: "0867720a79f8bfa9f8934a33ca202fb2"
server_key_iv: "c146cbe38ec4f7190574510c4704fa79"
server_key_version: "encryptionkey@0"
I20260928 21:12:19.892984 4319 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:19.907182 4319 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:19.910364 4319 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:19.910573 4319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:19.911158 4319 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:19.911981 4319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:19.912056 4319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.912124 4319 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:19.912158 4319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:19.921741 4319 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45673
I20260928 21:12:19.921777 4447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45673 every 8 connection(s)
I20260928 21:12:19.922683 4319 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:19.929771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4319
I20260928 21:12:19.929904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:19.930176 392 external_mini_cluster.cc:1428] Setting key 224d582053d29583d2b96019e00a0598
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:19.937278 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.924602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56585 (local address 127.0.98.62:36553)
0928 21:12:19.924748 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:19.924753 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:19.925763 (+ 1010us) server_negotiation.cc:407] Connection header received
0928 21:12:19.927163 (+ 1400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:19.927166 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:19.927213 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:19.927297 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:19.928479 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.929109 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.930687 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.931028 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.931145 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.933768 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:19.933774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:19.933782 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:19.933806 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:19.935119 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.935664 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.935667 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.935669 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.935750 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.936090 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.936095 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.936097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.936282 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:19.936425 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:19.936993 (+ 568us) server_negotiation.cc:299] Negotiation successful
0928 21:12:19.937116 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:12:19.938125 4450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.925101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36553 (local address 127.0.98.3:56585)
0928 21:12:19.925582 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:19.925633 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:19.926934 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:19.927462 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:19.927469 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:19.927831 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:19.928324 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:19.928337 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.929224 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.929240 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:19.930550 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:19.930559 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.931182 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.931186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:19.931361 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.931990 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:19.932007 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:19.933585 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:19.935250 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:19.935256 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:19.935273 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:19.935551 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:19.935848 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:19.935853 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:19.935855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:19.935994 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:19.936433 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:19.936442 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:19.936700 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 21:12:19.936942 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 21:12:19.937281 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"thread_start_us":112,"threads_started":1}
I20260928 21:12:19.939240 4448 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36553
I20260928 21:12:19.939471 4448 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:19.939920 4448 heartbeater.cc:511] Master 127.0.98.62:36553 requested a full tablet report, sending...
I20260928 21:12:19.940982 3989 ts_manager.cc:194] Registered new tserver with Master: 13a2c5026a774446b7e3696c09d489a3 (127.0.98.3:45673)
I20260928 21:12:19.941504 3989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56585
I20260928 21:12:19.944058 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:19.970212 4455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.960461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39704 (local address 127.0.98.1:37831)
0928 21:12:19.960735 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:19.960752 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:19.960779 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:12:19.960883 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:19.960896 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:19.961095 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:19.961239 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:19.961998 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.962845 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.963701 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.964111 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.964198 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.966687 (+ 2489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:19.966696 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:19.966710 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:19.966735 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:19.968308 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.968703 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.968708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.968710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.968822 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.969258 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.969261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.969263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.969547 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:19.969730 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:19.969924 (+ 194us) server_negotiation.cc:299] Negotiation successful
0928 21:12:19.970053 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":101,"threads_started":1}
Sep 28 21:12:19 dist-test-slave-v7xz krb5kdc[3944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629939, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:19.981769 4028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:19.973355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39010 (local address 127.0.98.62:36553)
0928 21:12:19.973490 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:19.973496 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:19.973561 (+ 65us) server_negotiation.cc:407] Connection header received
0928 21:12:19.973720 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:19.973723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:19.973776 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:19.973871 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:19.974526 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.975060 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.975860 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:19.976095 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:19.976209 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:19.978677 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:19.978685 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:19.978688 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:19.978715 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:19.980026 (+ 1311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.980476 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.980480 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.980481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.980563 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:19.980867 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:19.980870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:19.980872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:19.981102 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:19.981248 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:19.981481 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 21:12:19.981597 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 21:12:19.982213 3989 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39010
I20260928 21:12:19.982910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4051
I20260928 21:12:19.989684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4185
I20260928 21:12:19.995411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4319
I20260928 21:12:20.001034 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 3957
2026-09-28T21:12:20Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2268 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:20 dist-test-slave-v7xz krb5kdc[4460](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:20 dist-test-slave-v7xz krb5kdc[4460](info): set up 2 sockets
Sep 28 21:12:20 dist-test-slave-v7xz krb5kdc[4460](info): commencing operation
krb5kdc: starting...
W20260928 21:12:21.041116 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:21.167893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45563
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45563
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:21.276250 4473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:21.276547 4473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:21.276619 4473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:21.280622 4473 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:21.280705 4473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:21.280740 4473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:21.280779 4473 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:21.280813 4473 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:21.287321 4473 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45563
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45563
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:21.288347 4473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:21.289367 4473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:21.297372 4473 server_base.cc:1034] running on GCE node
W20260928 21:12:21.297433 4479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:21.297446 4481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:21.297371 4478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:21.298126 4473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:21.299252 4473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:21.300429 4473 hybrid_clock.cc:648] HybridClock initialized: now 1790629941300414 us; error 29 us; skew 500 ppm
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:21.303105 4473 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:21.303798 4473 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:21.303858 4473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:21.304016 4473 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:21.306000 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "30b63e1fac834955bc668ae67726d63e"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "d7bf82f9c454f53fe43e5c5f979b5f68"
server_key_iv: "d94efd56cb83cc8c0b690c7848cad26a"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.306509 4473 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "30b63e1fac834955bc668ae67726d63e"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "d7bf82f9c454f53fe43e5c5f979b5f68"
server_key_iv: "d94efd56cb83cc8c0b690c7848cad26a"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.310021 4473 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:21.312268 4487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.313623 4473 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:12:21.313753 4473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "30b63e1fac834955bc668ae67726d63e"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "d7bf82f9c454f53fe43e5c5f979b5f68"
server_key_iv: "d94efd56cb83cc8c0b690c7848cad26a"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.313861 4473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:21.324780 4473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:21.328043 4473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:21.328243 4473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:21.335024 4473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45563
I20260928 21:12:21.335312 4539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45563 every 8 connection(s)
I20260928 21:12:21.335723 4473 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:21.338768 4540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:21.343551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4473
I20260928 21:12:21.343703 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:21.343969 392 external_mini_cluster.cc:1428] Setting key fd95a8d3ee7edf15ce147675bdb17542
I20260928 21:12:21.344898 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e: Bootstrap starting.
I20260928 21:12:21.347445 4540 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:21.348275 4540 log.cc:824] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:21.350375 4540 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e: No bootstrap required, opened a new log
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:21.354341 4540 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b63e1fac834955bc668ae67726d63e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260928 21:12:21.354586 4540 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:21.354668 4540 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30b63e1fac834955bc668ae67726d63e, State: Initialized, Role: FOLLOWER
I20260928 21:12:21.355253 4540 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b63e1fac834955bc668ae67726d63e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260928 21:12:21.355443 4540 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:21.355516 4540 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:21.355624 4540 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:21.356894 4540 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b63e1fac834955bc668ae67726d63e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260928 21:12:21.357293 4540 leader_election.cc:302] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30b63e1fac834955bc668ae67726d63e; no voters:
I20260928 21:12:21.357689 4540 leader_election.cc:288] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:21.357808 4545 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:21.358083 4545 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [term 1 LEADER]: Becoming Leader. State: Replica: 30b63e1fac834955bc668ae67726d63e, State: Running, Role: LEADER
I20260928 21:12:21.358443 4545 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b63e1fac834955bc668ae67726d63e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } }
I20260928 21:12:21.359124 4540 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:21.359686 4546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30b63e1fac834955bc668ae67726d63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b63e1fac834955bc668ae67726d63e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } } }
I20260928 21:12:21.359722 4547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30b63e1fac834955bc668ae67726d63e. Latest consensus state: current_term: 1 leader_uuid: "30b63e1fac834955bc668ae67726d63e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30b63e1fac834955bc668ae67726d63e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45563 } } }
I20260928 21:12:21.359807 4546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:21.359819 4547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:21.359892 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:21.345415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35720 (local address 127.0.98.62:45563)
0928 21:12:21.345890 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:21.345907 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:21.345946 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:12:21.346669 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:21.346690 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:21.347067 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:21.347625 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:21.348538 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.349574 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.350441 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.350981 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.351150 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:21.354011 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:21.354017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:21.354033 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:21.354064 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:21.356239 (+ 2175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:21.356738 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:21.356746 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:21.356753 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:21.356950 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:21.357275 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:21.357278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:21.357280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:21.357952 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:21.358161 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:21.358467 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 21:12:21.358777 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":144,"threads_started":1}
I20260928 21:12:21.360154 4550 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:21.363535 4550 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:21.369764 4550 catalog_manager.cc:1347] Generated new cluster ID: 02aa2dbdc2fa4bc1bce125cf7c0d7885
I20260928 21:12:21.369868 4550 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:21.398324 4550 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:21.399720 4550 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:21.414773 4550 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e: Generated new TSK 0
I20260928 21:12:21.415664 4550 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:21.444584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:21.558699 4566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:21.558974 4566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:21.559052 4566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:21.563431 4566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:21.563501 4566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:21.563608 4566 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:21.569036 4566 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:21.570123 4566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:21.571142 4566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:21.580653 4574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:21.580663 4571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:21.580756 4572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:21.581064 4566 server_base.cc:1034] running on GCE node
I20260928 21:12:21.581501 4566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:21.582116 4566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:21.583299 4566 hybrid_clock.cc:648] HybridClock initialized: now 1790629941583287 us; error 30 us; skew 500 ppm
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:21.586020 4566 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:21.586757 4566 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:21.586843 4566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:21.587060 4566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:21.589095 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "2e85bf39dcb9451f9314eab4b06a6922"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "86a8174047d59df3f3c76b2820a16ffe"
server_key_iv: "ee2ed8f27dcf390442903d29e14c7907"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.589622 4566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2e85bf39dcb9451f9314eab4b06a6922"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "86a8174047d59df3f3c76b2820a16ffe"
server_key_iv: "ee2ed8f27dcf390442903d29e14c7907"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.593185 4566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:21.595419 4580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.596587 4566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:21.596716 4566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "2e85bf39dcb9451f9314eab4b06a6922"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "86a8174047d59df3f3c76b2820a16ffe"
server_key_iv: "ee2ed8f27dcf390442903d29e14c7907"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.596879 4566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:21.615388 4566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:21.618578 4566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:21.618780 4566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:21.619345 4566 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:21.620154 4566 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:21.620219 4566 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.620282 4566 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:21.620326 4566 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.629346 4566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44431
I20260928 21:12:21.629361 4693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44431 every 8 connection(s)
I20260928 21:12:21.629958 4566 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:21.631042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4566
I20260928 21:12:21.631162 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:21.631439 392 external_mini_cluster.cc:1428] Setting key ac823d6a6dffb7d9d9ed41020a8b45d4
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:21.646461 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:21.631809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43175 (local address 127.0.98.62:45563)
0928 21:12:21.631952 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:21.631956 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:21.633073 (+ 1117us) server_negotiation.cc:407] Connection header received
0928 21:12:21.635131 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:21.635137 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:21.635203 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:21.635311 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:21.636552 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.637229 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.639185 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.639422 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.639507 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:21.642508 (+ 3001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:21.642513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:21.642518 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:21.642554 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:21.643968 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:21.644565 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:21.644567 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:21.644568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:21.644642 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:21.645071 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:21.645074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:21.645076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:21.645345 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:21.645461 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:21.646205 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 21:12:21.646319 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:12:21.647701 4696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:21.632326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45563 (local address 127.0.98.1:43175)
0928 21:12:21.632915 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:21.632986 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:21.634877 (+ 1891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:21.635495 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:21.635503 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:21.635865 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:21.636394 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:21.636408 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.637365 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.637372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:21.639044 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:21.639060 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.639596 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.639603 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:21.639864 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:21.640555 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:21.640583 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:21.642290 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:21.644105 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:21.644110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:21.644124 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:21.644403 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:21.644723 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:21.644725 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:21.644726 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:21.644938 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:21.645502 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:21.645512 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:21.645915 (+ 403us) client_negotiation.cc:769] Sending connection context
0928 21:12:21.646157 (+ 242us) client_negotiation.cc:240] Negotiation successful
0928 21:12:21.646605 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":137,"threads_started":1}
I20260928 21:12:21.649120 4694 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45563
I20260928 21:12:21.649400 4694 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:21.650025 4694 heartbeater.cc:511] Master 127.0.98.62:45563 requested a full tablet report, sending...
I20260928 21:12:21.651888 4504 ts_manager.cc:194] Registered new tserver with Master: 2e85bf39dcb9451f9314eab4b06a6922 (127.0.98.1:44431)
I20260928 21:12:21.653151 4504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:21.662303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:21.769294 4699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:21.769578 4699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:21.769652 4699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:21.773665 4699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:21.773738 4699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:21.773837 4699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:21.779196 4699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:21.780203 4699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:21.781239 4699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:21.790307 4707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:21.790316 4705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:21.790311 4704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:21.790649 4699 server_base.cc:1034] running on GCE node
I20260928 21:12:21.791152 4699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:21.791827 4699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:21.793012 4699 hybrid_clock.cc:648] HybridClock initialized: now 1790629941792994 us; error 36 us; skew 500 ppm
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:21.795953 4699 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:21.796720 4699 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:21.796826 4699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:21.797078 4699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:21.799132 4699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "3367d79eeae8b5cd808004d0e7065dd9"
server_key_iv: "0bcd90eca2a0d7202b81c819159991e7"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.799708 4699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "3367d79eeae8b5cd808004d0e7065dd9"
server_key_iv: "0bcd90eca2a0d7202b81c819159991e7"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.803507 4699 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:21.805934 4713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.807255 4699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:21.807381 4699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e"
format_stamp: "Formatted at 2026-09-28 21:12:21 on dist-test-slave-v7xz"
server_key: "3367d79eeae8b5cd808004d0e7065dd9"
server_key_iv: "0bcd90eca2a0d7202b81c819159991e7"
server_key_version: "encryptionkey@0"
I20260928 21:12:21.807492 4699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:21.827127 4699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:21.830458 4699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:21.830677 4699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:21.831266 4699 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:21.832020 4699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:21.832067 4699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.832106 4699 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:21.832127 4699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:21.840764 4699 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44755
I20260928 21:12:21.840775 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44755 every 8 connection(s)
I20260928 21:12:21.841395 4699 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:21.848349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4699
I20260928 21:12:21.848493 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:21.848764 392 external_mini_cluster.cc:1428] Setting key 194dfdb4c0c29fe7aaaa2efacd2c77f3
Sep 28 21:12:21 dist-test-slave-v7xz krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629941, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:21.861198 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:21.843057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38587 (local address 127.0.98.62:45563)
0928 21:12:21.843193 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:21.843196 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:21.844086 (+ 890us) server_negotiation.cc:407] Connection header received
0928 21:12:21.845413 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:21.845417 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:21.845463 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:21.845544 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:21.846706 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.847209 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.848319 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.848618 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.848735 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:21.857326 (+ 8591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:21.857329 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:21.857331 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:21.857354 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:21.858853 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:21.859428 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:21.859432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:21.859433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:21.859557 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:21.859933 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:21.859938 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:21.859939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:21.860242 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:21.860364 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:21.860876 (+ 512us) server_negotiation.cc:299] Negotiation successful
0928 21:12:21.860995 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:12:21.862040 4829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:21.843476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45563 (local address 127.0.98.2:38587)
0928 21:12:21.843936 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:21.843977 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:21.845213 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:21.845734 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:21.845741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:21.846115 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:21.846582 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:21.846595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.847308 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.847310 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:21.848185 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:21.848193 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:21.848773 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:21.848777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:21.848996 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:21.855341 (+ 6345us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:21.855358 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:21.857144 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:21.859026 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:21.859036 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:21.859056 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:21.859323 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:21.859655 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:21.859659 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:21.859661 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:21.859834 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:21.860361 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:21.860368 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:21.860619 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 21:12:21.860806 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:12:21.861175 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":104,"threads_started":1}
I20260928 21:12:21.863246 4827 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45563
I20260928 21:12:21.863554 4827 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:21.864108 4827 heartbeater.cc:511] Master 127.0.98.62:45563 requested a full tablet report, sending...
I20260928 21:12:21.865320 4504 ts_manager.cc:194] Registered new tserver with Master: a0fedaf7df5a4b039e17ba1c3ebb835e (127.0.98.2:44755)
I20260928 21:12:21.865978 4504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38587
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:21.873906 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:21.982306 4832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:21.982607 4832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:21.982684 4832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:21.986696 4832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:21.986790 4832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:21.986886 4832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:21.992091 4832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4832
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:21.993192 4832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:21.994218 4832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:22.003844 4840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:22.003945 4838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:22.003940 4837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:22.004082 4832 server_base.cc:1034] running on GCE node
I20260928 21:12:22.004618 4832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:22.005316 4832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:22.006474 4832 hybrid_clock.cc:648] HybridClock initialized: now 1790629942006442 us; error 47 us; skew 500 ppm
Sep 28 21:12:22 dist-test-slave-v7xz krb5kdc[4460](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:22.009333 4832 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:22.010016 4832 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:22.010102 4832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:22.010330 4832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:22.012426 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "67bf2e5ea977488c9fa8fd5998be084d"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "f46ead38c890493776ab98a28a02547c"
server_key_iv: "55cdc456feb71ba7f82938147577d55d"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.013019 4832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "67bf2e5ea977488c9fa8fd5998be084d"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "f46ead38c890493776ab98a28a02547c"
server_key_iv: "55cdc456feb71ba7f82938147577d55d"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.017270 4832 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:12:22.019776 4846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.021155 4832 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:22.021277 4832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "67bf2e5ea977488c9fa8fd5998be084d"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "f46ead38c890493776ab98a28a02547c"
server_key_iv: "55cdc456feb71ba7f82938147577d55d"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.021386 4832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:22.041601 4832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:22.045236 4832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:22.045449 4832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:22.046067 4832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:22.046895 4832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:22.046941 4832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.047013 4832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:22.047047 4832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.055634 4832 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40161
I20260928 21:12:22.055653 4959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40161 every 8 connection(s)
I20260928 21:12:22.056241 4832 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:22.059772 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 4832
I20260928 21:12:22.059922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:22.060254 392 external_mini_cluster.cc:1428] Setting key de448712e2ba631d5c81b288a0287e56
Sep 28 21:12:22 dist-test-slave-v7xz krb5kdc[4460](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629942, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:22.070812 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.057966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34299 (local address 127.0.98.62:45563)
0928 21:12:22.058243 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.058248 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.059024 (+ 776us) server_negotiation.cc:407] Connection header received
0928 21:12:22.060942 (+ 1918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.060946 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.061016 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.061111 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:22.062706 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.063263 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.064370 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.064576 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.064657 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.067192 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:22.067196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:22.067198 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:22.067223 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:22.068696 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:22.069363 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:22.069366 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:22.069367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:22.069449 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:22.069754 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:22.069756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:22.069757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:22.069975 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:22.070055 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.070576 (+ 521us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.070690 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260928 21:12:22.071641 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.058417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45563 (local address 127.0.98.3:34299)
0928 21:12:22.058867 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.058911 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.060639 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.061336 (+ 697us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.061346 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.061912 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:22.062552 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.062574 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.063366 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.063368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.064256 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.064264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.064701 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.064704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.064893 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.065541 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:22.065557 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:22.067035 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:22.068971 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:22.068976 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:22.068989 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:22.069249 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:22.069535 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:22.069538 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:22.069539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:22.069674 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:22.070090 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:22.070096 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:22.070338 (+ 242us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.070519 (+ 181us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.070792 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260928 21:12:22.072788 4960 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45563
I20260928 21:12:22.073079 4960 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:22.073576 4960 heartbeater.cc:511] Master 127.0.98.62:45563 requested a full tablet report, sending...
I20260928 21:12:22.074651 4504 ts_manager.cc:194] Registered new tserver with Master: 67bf2e5ea977488c9fa8fd5998be084d (127.0.98.3:40161)
I20260928 21:12:22.075246 4504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34299
I20260928 21:12:22.086202 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:12:22.095451 4543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.088133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35732 (local address 127.0.98.62:45563)
0928 21:12:22.088310 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.088314 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.088458 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:12:22.088610 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.088613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.088659 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.088743 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:22.089512 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.090027 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.090855 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.091071 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.091175 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.092488 (+ 1313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:22.092492 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:22.092495 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:22.092523 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:22.093795 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:22.094243 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:22.094248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:22.094250 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:22.094327 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:22.094588 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:22.094594 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:22.094597 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:22.094814 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:22.094952 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.095178 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.095290 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260928 21:12:22.098125 4504 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35732:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:22.101157 4504 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:22.115621 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.110238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55864 (local address 127.0.98.2:44755)
0928 21:12:22.110561 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.110574 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.110597 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:12:22.110673 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.110680 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.110911 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.111065 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:22.111515 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.112724 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.114311 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.114960 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.115080 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.115227 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.115358 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.115451 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233,"thread_start_us":153,"threads_started":1}
I20260928 21:12:22.115622 4971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.109110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44755 (local address 127.0.0.1:55864)
0928 21:12:22.109424 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.109489 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.109610 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.111056 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.111062 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.111084 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:22.111335 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.111343 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.112874 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.112877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.114195 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.114203 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.115079 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.115083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.115185 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.115220 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.115331 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.115451 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":25,"thread_start_us":77,"threads_started":1}
I20260928 21:12:22.115895 4972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.109435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40161 (local address 127.0.0.1:41442)
0928 21:12:22.109946 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.109962 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.110036 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.112062 (+ 2026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.112065 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.112084 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:22.112162 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.112167 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.113338 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.113345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.114483 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.114494 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.115532 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.115535 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.115609 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.115629 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.115680 (+ 51us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.115747 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"mutex_wait_us":31,"spinlock_wait_cycles":1408,"thread_start_us":83,"threads_started":1}
I20260928 21:12:22.116176 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.109156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44431 (local address 127.0.0.1:56396)
0928 21:12:22.109919 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.109932 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.110036 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.111242 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.111244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.111263 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:22.111361 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.111369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.112639 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.112645 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.114103 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.114113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.115292 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.115296 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.115394 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.115433 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.115504 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.115575 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":28,"spinlock_wait_cycles":8320,"thread_start_us":109,"threads_started":1}
I20260928 21:12:22.116530 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.109878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41442 (local address 127.0.98.3:40161)
0928 21:12:22.111555 (+ 1677us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.111567 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.111586 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:12:22.111737 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.111742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.111921 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.112068 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:22.112285 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.113212 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.114613 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.115378 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.116073 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.116221 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.116281 (+ 60us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.116370 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1575,"thread_start_us":93,"threads_started":1}
I20260928 21:12:22.116575 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.109928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56396 (local address 127.0.98.1:44431)
0928 21:12:22.110836 (+ 908us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.110843 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.110865 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:12:22.110935 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.110940 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.111117 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.111251 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:22.111512 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.112520 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.114246 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.114987 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.115089 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.115228 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.116336 (+ 1108us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.116429 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":62,"threads_started":1}
I20260928 21:12:22.118969 4761 tablet_service.cc:1467] Processing CreateTablet for tablet 3438a603a406494784f08c54049ec2be (DEFAULT_TABLE table=test-table [id=54e2e41acc784e4ab512c20588838d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:22.119565 4894 tablet_service.cc:1467] Processing CreateTablet for tablet 3438a603a406494784f08c54049ec2be (DEFAULT_TABLE table=test-table [id=54e2e41acc784e4ab512c20588838d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:22.120152 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3438a603a406494784f08c54049ec2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:22.119853 4628 tablet_service.cc:1467] Processing CreateTablet for tablet 3438a603a406494784f08c54049ec2be (DEFAULT_TABLE table=test-table [id=54e2e41acc784e4ab512c20588838d69]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:22.120525 4894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3438a603a406494784f08c54049ec2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:22.120860 4628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3438a603a406494784f08c54049ec2be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:22.127030 4977 tablet_bootstrap.cc:492] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Bootstrap starting.
I20260928 21:12:22.128219 4978 tablet_bootstrap.cc:492] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: Bootstrap starting.
I20260928 21:12:22.129180 4979 tablet_bootstrap.cc:492] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Bootstrap starting.
I20260928 21:12:22.129324 4977 tablet_bootstrap.cc:654] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:22.130304 4978 tablet_bootstrap.cc:654] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:22.130399 4977 log.cc:824] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:22.131315 4979 tablet_bootstrap.cc:654] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:22.131551 4978 log.cc:824] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:22.132429 4977 tablet_bootstrap.cc:492] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: No bootstrap required, opened a new log
I20260928 21:12:22.132514 4979 log.cc:824] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:22.132653 4977 ts_tablet_manager.cc:1397] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:12:22.133597 4978 tablet_bootstrap.cc:492] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: No bootstrap required, opened a new log
I20260928 21:12:22.133785 4978 ts_tablet_manager.cc:1397] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.004s
I20260928 21:12:22.134387 4979 tablet_bootstrap.cc:492] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: No bootstrap required, opened a new log
I20260928 21:12:22.134562 4979 ts_tablet_manager.cc:1397] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 21:12:22.135915 4977 raft_consensus.cc:348] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.136171 4977 raft_consensus.cc:374] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:22.136224 4977 raft_consensus.cc:729] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bf2e5ea977488c9fa8fd5998be084d, State: Initialized, Role: FOLLOWER
I20260928 21:12:22.136663 4977 consensus_queue.cc:260] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.137285 4960 heartbeater.cc:503] Master 127.0.98.62:45563 was elected leader, sending a full tablet report...
I20260928 21:12:22.137499 4978 raft_consensus.cc:348] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.137761 4978 raft_consensus.cc:374] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:22.137820 4978 raft_consensus.cc:729] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e85bf39dcb9451f9314eab4b06a6922, State: Initialized, Role: FOLLOWER
I20260928 21:12:22.138226 4978 consensus_queue.cc:260] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.138218 4979 raft_consensus.cc:348] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.138484 4979 raft_consensus.cc:374] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:22.138535 4979 raft_consensus.cc:729] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0fedaf7df5a4b039e17ba1c3ebb835e, State: Initialized, Role: FOLLOWER
I20260928 21:12:22.138649 4977 ts_tablet_manager.cc:1428] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260928 21:12:22.138945 4979 consensus_queue.cc:260] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.139151 4978 ts_tablet_manager.cc:1428] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:12:22.139460 4694 heartbeater.cc:503] Master 127.0.98.62:45563 was elected leader, sending a full tablet report...
I20260928 21:12:22.139588 4827 heartbeater.cc:503] Master 127.0.98.62:45563 was elected leader, sending a full tablet report...
I20260928 21:12:22.139811 4979 ts_tablet_manager.cc:1428] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Time spent starting tablet: real 0.005s user 0.002s sys 0.004s
W20260928 21:12:22.307140 4961 tablet.cc:2369] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:22.342422 4828 tablet.cc:2369] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:22.381145 4695 tablet.cc:2369] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:22.381487 4983 raft_consensus.cc:482] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:22.381687 4983 raft_consensus.cc:504] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.382838 4983 leader_election.cc:288] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a0fedaf7df5a4b039e17ba1c3ebb835e (127.0.98.2:44755), 2e85bf39dcb9451f9314eab4b06a6922 (127.0.98.1:44431)
I20260928 21:12:22.386847 4975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.383232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53421 (local address 127.0.98.1:44431)
0928 21:12:22.383377 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.383383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.383462 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:12:22.383588 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.383594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.383652 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.383730 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:22.384176 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.384644 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.385824 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.386308 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.386414 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.386475 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.386631 (+ 156us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.386710 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:12:22.386847 4962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.383103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44431 (local address 127.0.98.3:53421)
0928 21:12:22.383373 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.383390 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.383489 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.383950 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.383952 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.383965 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:22.384070 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.384078 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.384769 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.384772 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.385692 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.385701 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.386429 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.386432 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.386506 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.386521 (+ 15us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.386614 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.386714 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":62}
I20260928 21:12:22.386835 4976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.383144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32779 (local address 127.0.98.2:44755)
0928 21:12:22.383312 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.383318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.383646 (+ 328us) server_negotiation.cc:407] Connection header received
0928 21:12:22.383745 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.383747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.383791 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.383852 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:22.384097 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.384627 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.385829 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.386228 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.386309 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.386355 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.386626 (+ 271us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.386704 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:12:22.386862 4986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.383114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44755 (local address 127.0.98.3:32779)
0928 21:12:22.383546 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.383561 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.383660 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.383885 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.383887 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.383899 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:22.383992 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.383996 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.384737 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.384740 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.385694 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:22.385703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.386334 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.386340 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:22.386447 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.386478 (+ 31us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.386600 (+ 122us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.386701 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":81,"thread_start_us":115,"threads_started":1}
I20260928 21:12:22.387544 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438a603a406494784f08c54049ec2be" candidate_uuid: "67bf2e5ea977488c9fa8fd5998be084d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" is_pre_election: true
I20260928 21:12:22.387547 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438a603a406494784f08c54049ec2be" candidate_uuid: "67bf2e5ea977488c9fa8fd5998be084d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e85bf39dcb9451f9314eab4b06a6922" is_pre_election: true
I20260928 21:12:22.387864 4781 raft_consensus.cc:2454] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bf2e5ea977488c9fa8fd5998be084d in term 0.
I20260928 21:12:22.387864 4648 raft_consensus.cc:2454] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67bf2e5ea977488c9fa8fd5998be084d in term 0.
I20260928 21:12:22.388356 4848 leader_election.cc:302] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e85bf39dcb9451f9314eab4b06a6922, 67bf2e5ea977488c9fa8fd5998be084d; no voters:
I20260928 21:12:22.388607 4983 raft_consensus.cc:2781] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:22.388697 4983 raft_consensus.cc:482] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:22.388741 4983 raft_consensus.cc:3037] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:22.389783 4983 raft_consensus.cc:504] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.390143 4983 leader_election.cc:288] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [CANDIDATE]: Term 1 election: Requested vote from peers a0fedaf7df5a4b039e17ba1c3ebb835e (127.0.98.2:44755), 2e85bf39dcb9451f9314eab4b06a6922 (127.0.98.1:44431)
I20260928 21:12:22.390523 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438a603a406494784f08c54049ec2be" candidate_uuid: "67bf2e5ea977488c9fa8fd5998be084d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e"
I20260928 21:12:22.390599 4648 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3438a603a406494784f08c54049ec2be" candidate_uuid: "67bf2e5ea977488c9fa8fd5998be084d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e85bf39dcb9451f9314eab4b06a6922"
I20260928 21:12:22.390645 4781 raft_consensus.cc:3037] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:22.390779 4648 raft_consensus.cc:3037] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:22.391700 4781 raft_consensus.cc:2454] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bf2e5ea977488c9fa8fd5998be084d in term 1.
I20260928 21:12:22.391872 4648 raft_consensus.cc:2454] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67bf2e5ea977488c9fa8fd5998be084d in term 1.
I20260928 21:12:22.392040 4847 leader_election.cc:302] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67bf2e5ea977488c9fa8fd5998be084d, a0fedaf7df5a4b039e17ba1c3ebb835e; no voters:
I20260928 21:12:22.392280 4983 raft_consensus.cc:2781] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:22.392484 4983 raft_consensus.cc:686] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 1 LEADER]: Becoming Leader. State: Replica: 67bf2e5ea977488c9fa8fd5998be084d, State: Running, Role: LEADER
I20260928 21:12:22.392827 4983 consensus_queue.cc:237] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } }
I20260928 21:12:22.395985 4502 catalog_manager.cc:5495] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d reported cstate change: term changed from 0 to 1, leader changed from <none> to 67bf2e5ea977488c9fa8fd5998be084d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "67bf2e5ea977488c9fa8fd5998be084d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67bf2e5ea977488c9fa8fd5998be084d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40161 } health_report { overall_health: HEALTHY } } }
I20260928 21:12:22.408599 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.405179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41450 (local address 127.0.98.3:40161)
0928 21:12:22.405289 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.405293 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.405306 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:22.405417 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.405419 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.405481 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.405595 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:22.405850 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.406330 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.407173 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:22.407384 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:22.407504 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:22.407815 (+ 311us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:22.408208 (+ 393us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:22.408339 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.408379 (+ 40us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.408430 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260928 21:12:22.414641 4781 raft_consensus.cc:1261] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 1 FOLLOWER]: Refusing update from remote peer 67bf2e5ea977488c9fa8fd5998be084d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:22.414642 4648 raft_consensus.cc:1261] T 3438a603a406494784f08c54049ec2be P 2e85bf39dcb9451f9314eab4b06a6922 [term 1 FOLLOWER]: Refusing update from remote peer 67bf2e5ea977488c9fa8fd5998be084d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:22.415602 4987 consensus_queue.cc:1035] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e85bf39dcb9451f9314eab4b06a6922" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:22.415781 4983 consensus_queue.cc:1035] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0fedaf7df5a4b039e17ba1c3ebb835e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:22.423341 4990 mvcc.cc:205] Tried to move back new op lower bound from 7334420244123435008 to 7334420244043747328. Current Snapshot: MvccSnapshot[applied={T|T < 7334420244123435008}]
I20260928 21:12:22.425355 4991 mvcc.cc:205] Tried to move back new op lower bound from 7334420244123435008 to 7334420244043747328. Current Snapshot: MvccSnapshot[applied={T|T < 7334420244123435008}]
I20260928 21:12:22.432557 4504 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:22.432744 4504 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35732:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:22.435060 4504 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 30b63e1fac834955bc668ae67726d63e: Sending DeleteTablet for 3 replicas of tablet 3438a603a406494784f08c54049ec2be
I20260928 21:12:22.435802 4628 tablet_service.cc:1514] Processing DeleteTablet for tablet 3438a603a406494784f08c54049ec2be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56396
I20260928 21:12:22.435849 4761 tablet_service.cc:1514] Processing DeleteTablet for tablet 3438a603a406494784f08c54049ec2be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55864
I20260928 21:12:22.435860 4894 tablet_service.cc:1514] Processing DeleteTablet for tablet 3438a603a406494784f08c54049ec2be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:22 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41442
I20260928 21:12:22.436342 5001 tablet_replica.cc:330] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: stopping tablet replica
I20260928 21:12:22.436663 5001 raft_consensus.cc:2229] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:22.437041 5001 raft_consensus.cc:2258] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:22.437085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4566
I20260928 21:12:22.438263 4999 tablet_replica.cc:330] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: stopping tablet replica
I20260928 21:12:22.438556 4999 raft_consensus.cc:2229] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:22.438891 4999 raft_consensus.cc:2258] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:22.439085 5001 ts_tablet_manager.cc:1905] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:22.440241 4999 ts_tablet_manager.cc:1905] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:22.442530 5001 ts_tablet_manager.cc:1918] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:22.442694 5001 log.cc:1196] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/3438a603a406494784f08c54049ec2be
I20260928 21:12:22.443136 5001 ts_tablet_manager.cc:1939] T 3438a603a406494784f08c54049ec2be P 67bf2e5ea977488c9fa8fd5998be084d: Deleting consensus metadata
I20260928 21:12:22.443157 4999 ts_tablet_manager.cc:1918] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:22.443249 4999 log.cc:1196] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/3438a603a406494784f08c54049ec2be
I20260928 21:12:22.443588 4999 ts_tablet_manager.cc:1939] T 3438a603a406494784f08c54049ec2be P a0fedaf7df5a4b039e17ba1c3ebb835e: Deleting consensus metadata
I20260928 21:12:22.444612 4491 catalog_manager.cc:4841] TS 67bf2e5ea977488c9fa8fd5998be084d (127.0.98.3:40161): tablet 3438a603a406494784f08c54049ec2be (table test-table [id=54e2e41acc784e4ab512c20588838d69]) successfully deleted
I20260928 21:12:22.444636 4488 catalog_manager.cc:4841] TS a0fedaf7df5a4b039e17ba1c3ebb835e (127.0.98.2:44755): tablet 3438a603a406494784f08c54049ec2be (table test-table [id=54e2e41acc784e4ab512c20588838d69]) successfully deleted
W20260928 21:12:22.445986 4489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44431 (error 108)
I20260928 21:12:22.446179 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4699
I20260928 21:12:22.447515 4973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.447247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44431 (local address 127.0.0.1:56408)
0928 21:12:22.447377 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.447445 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44431: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:12:22.447729 4489 catalog_manager.cc:4584] TS 2e85bf39dcb9451f9314eab4b06a6922 (127.0.98.1:44431): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3438a603a406494784f08c54049ec2be: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44431: connect: Connection refused (error 111)
I20260928 21:12:22.454452 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4832
I20260928 21:12:22.460496 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 4473
2026-09-28T21:12:22Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2459 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-28T21:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:22Z Disabled control of system clock
I20260928 21:12:22.490269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46267
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46267
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:12:22.606534 5006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:22.606829 5006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:22.606870 5006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:22.616694 5006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:22.616786 5006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:22.616802 5006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:22.616819 5006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:22.616863 5006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:22.622390 5006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46267
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5006
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:22.623605 5006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:22.625272 5006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:22.634548 5011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:22.634531 5014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:22.634603 5006 server_base.cc:1034] running on GCE node
W20260928 21:12:22.634548 5012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:22.635383 5006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:22.636775 5006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:22.638000 5006 hybrid_clock.cc:648] HybridClock initialized: now 1790629942637965 us; error 48 us; skew 500 ppm
I20260928 21:12:22.640275 5006 webserver.cc:460] Webserver started at http://127.0.98.62:35129/ using document root <none> and password file <none>
I20260928 21:12:22.641032 5006 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:22.641146 5006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:22.641372 5006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:22.643587 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "9c2c613b8ea14a90816a598729663d82"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "fda44cfd7f86060d43cbecfccf243c3e"
server_key_iv: "a6a913ec145514e46a7a5f193b8f8414"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.644189 5006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c2c613b8ea14a90816a598729663d82"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "fda44cfd7f86060d43cbecfccf243c3e"
server_key_iv: "a6a913ec145514e46a7a5f193b8f8414"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.648614 5006 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260928 21:12:22.652002 5020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.653716 5006 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:22.653892 5006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "9c2c613b8ea14a90816a598729663d82"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "fda44cfd7f86060d43cbecfccf243c3e"
server_key_iv: "a6a913ec145514e46a7a5f193b8f8414"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.654031 5006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:22.659121 5006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:22.659878 5006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:22.660094 5006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:22.668110 5006 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46267
I20260928 21:12:22.668111 5072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46267 every 8 connection(s)
I20260928 21:12:22.669373 5006 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:22.672708 5073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:22.675665 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5006
I20260928 21:12:22.675797 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:22.676070 392 external_mini_cluster.cc:1428] Setting key d78e66d755ac2c2769e1c6d6e50e1614
I20260928 21:12:22.678772 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: Bootstrap starting.
I20260928 21:12:22.681308 5073 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:22.682060 5073 log.cc:824] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:22.682436 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.677577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42408 (local address 127.0.98.62:46267)
0928 21:12:22.678264 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.678287 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.678336 (+ 49us) server_negotiation.cc:407] Connection header received
0928 21:12:22.679268 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.679290 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.679678 (+ 388us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.679980 (+ 302us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:22.680909 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:22.680914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:22.680926 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:22.680938 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:22.681002 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:22.681020 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:22.681117 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.681301 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.681583 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":165,"threads_started":1}
I20260928 21:12:22.684353 5073 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: No bootstrap required, opened a new log
I20260928 21:12:22.687750 5073 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c613b8ea14a90816a598729663d82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260928 21:12:22.687961 5073 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:22.688002 5073 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c2c613b8ea14a90816a598729663d82, State: Initialized, Role: FOLLOWER
I20260928 21:12:22.688387 5073 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c613b8ea14a90816a598729663d82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260928 21:12:22.688503 5073 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:22.688546 5073 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:22.688616 5073 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:22.707686 5073 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c613b8ea14a90816a598729663d82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260928 21:12:22.708221 5073 leader_election.cc:302] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9c2c613b8ea14a90816a598729663d82; no voters:
I20260928 21:12:22.708603 5073 leader_election.cc:288] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:22.708755 5078 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:22.709106 5078 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [term 1 LEADER]: Becoming Leader. State: Replica: 9c2c613b8ea14a90816a598729663d82, State: Running, Role: LEADER
I20260928 21:12:22.709522 5078 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c613b8ea14a90816a598729663d82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } }
I20260928 21:12:22.710170 5073 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:22.711454 5080 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9c2c613b8ea14a90816a598729663d82. Latest consensus state: current_term: 1 leader_uuid: "9c2c613b8ea14a90816a598729663d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c613b8ea14a90816a598729663d82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } } }
I20260928 21:12:22.711620 5080 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:22.712050 5079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9c2c613b8ea14a90816a598729663d82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9c2c613b8ea14a90816a598729663d82" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46267 } } }
I20260928 21:12:22.712172 5079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82 [sys.catalog]: This master's current role is: LEADER
W20260928 21:12:22.715601 5093 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:12:22.715698 5093 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:12:22.715867 5088 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:22.716683 5088 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:22.722473 5088 catalog_manager.cc:1347] Generated new cluster ID: c4c0605024494c3a8a75b840b63a2054
I20260928 21:12:22.722565 5088 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:22.745467 5088 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:22.745884 5088 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:22.752287 5088 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: Generated new TSK 0
I20260928 21:12:22.753099 5088 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:22.763761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--builtin_ntp_servers=127.0.98.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:12:22.910578 5097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:22.910871 5097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:22.910913 5097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:22.915659 5097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:22.915730 5097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:22.915813 5097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:22.920878 5097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:22.922027 5097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:22.923496 5097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:22.933694 5102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:22.933694 5105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:22.933704 5103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:22.933975 5097 server_base.cc:1034] running on GCE node
I20260928 21:12:22.934433 5097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:22.935056 5097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:22.936216 5097 hybrid_clock.cc:648] HybridClock initialized: now 1790629942936202 us; error 31 us; skew 500 ppm
I20260928 21:12:22.938827 5097 webserver.cc:460] Webserver started at http://127.0.98.1:45997/ using document root <none> and password file <none>
I20260928 21:12:22.939399 5097 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:22.939469 5097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:22.939642 5097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:22.941811 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "3c5311b75deb4d2ebca9d7779a0d90f4"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "98dfc03dd1047e01169ba81231cce5cb"
server_key_iv: "c3de728dfe6affd5014e254413f067ea"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.942337 5097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3c5311b75deb4d2ebca9d7779a0d90f4"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "98dfc03dd1047e01169ba81231cce5cb"
server_key_iv: "c3de728dfe6affd5014e254413f067ea"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.946235 5097 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:22.948715 5111 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.950047 5097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:22.950168 5097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "3c5311b75deb4d2ebca9d7779a0d90f4"
format_stamp: "Formatted at 2026-09-28 21:12:22 on dist-test-slave-v7xz"
server_key: "98dfc03dd1047e01169ba81231cce5cb"
server_key_iv: "c3de728dfe6affd5014e254413f067ea"
server_key_version: "encryptionkey@0"
I20260928 21:12:22.950284 5097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:22.954934 5097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:22.955536 5097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:22.955721 5097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:22.956262 5097 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:22.957219 5097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:22.957286 5097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.957350 5097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:22.957391 5097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:22.966291 5097 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35557
I20260928 21:12:22.966313 5224 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35557 every 8 connection(s)
I20260928 21:12:22.967249 5097 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:22.971180 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5097
I20260928 21:12:22.971325 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:22.971663 392 external_mini_cluster.cc:1428] Setting key b2f5ea17fb2e542b3cb182381be6cfe1
I20260928 21:12:22.974000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--builtin_ntp_servers=127.0.98.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260928 21:12:22.974166 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.969148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47925 (local address 127.0.98.62:46267)
0928 21:12:22.969298 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:22.969303 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:22.970515 (+ 1212us) server_negotiation.cc:407] Connection header received
0928 21:12:22.972200 (+ 1685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:22.972206 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:22.972257 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:22.972334 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:22.973669 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:22.973672 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:22.973674 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:22.973686 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:22.973707 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:22.973715 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:22.973789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:22.973978 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:12:22.974015 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:12:22.975399 5227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:22.969615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46267 (local address 127.0.98.1:47925)
0928 21:12:22.970365 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:22.970404 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:22.971957 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:22.972542 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:22.972549 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:22.972642 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:22.973356 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:22.973358 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:22.973412 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:22.973505 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:22.973804 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:22.973812 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:22.973822 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:12:22.974014 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 21:12:22.974328 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260928 21:12:22.976939 5225 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46267
I20260928 21:12:22.977190 5225 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:22.977663 5225 heartbeater.cc:511] Master 127.0.98.62:46267 requested a full tablet report, sending...
I20260928 21:12:22.979523 5037 ts_manager.cc:194] Registered new tserver with Master: 3c5311b75deb4d2ebca9d7779a0d90f4 (127.0.98.1:35557)
W20260928 21:12:23.087721 5228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:23.087986 5228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:23.088025 5228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:23.091931 5228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:23.091991 5228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:23.092047 5228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:23.096781 5228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:23.097857 5228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:23.099165 5228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:23.108567 5234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:23.108554 5233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:23.108577 5236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:23.108740 5228 server_base.cc:1034] running on GCE node
I20260928 21:12:23.109176 5228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:23.109869 5228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:23.111063 5228 hybrid_clock.cc:648] HybridClock initialized: now 1790629943111052 us; error 34 us; skew 500 ppm
I20260928 21:12:23.113219 5228 webserver.cc:460] Webserver started at http://127.0.98.2:40053/ using document root <none> and password file <none>
I20260928 21:12:23.113797 5228 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:23.113884 5228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:23.114113 5228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:23.116185 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "358a628e3aea4a10b9f4bbc6cfbb8483"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "1e6f46a684d772b5b8e776bb96ae8331"
server_key_iv: "2ce6142e8b09fef8873aa3ae6533a47c"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.116752 5228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "358a628e3aea4a10b9f4bbc6cfbb8483"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "1e6f46a684d772b5b8e776bb96ae8331"
server_key_iv: "2ce6142e8b09fef8873aa3ae6533a47c"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.120424 5228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:23.122939 5242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.124140 5228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:23.124269 5228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "358a628e3aea4a10b9f4bbc6cfbb8483"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "1e6f46a684d772b5b8e776bb96ae8331"
server_key_iv: "2ce6142e8b09fef8873aa3ae6533a47c"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.124379 5228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:23.129077 5228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:23.129690 5228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:23.129866 5228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:23.130450 5228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:23.131352 5228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:23.131419 5228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.131482 5228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:23.131531 5228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.140220 5228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45961
I20260928 21:12:23.140239 5355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45961 every 8 connection(s)
I20260928 21:12:23.141225 5228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:23.147380 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.143036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47547 (local address 127.0.98.62:46267)
0928 21:12:23.143193 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.143197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.144306 (+ 1109us) server_negotiation.cc:407] Connection header received
0928 21:12:23.145788 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.145791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.145827 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.145905 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.147030 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.147033 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.147035 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.147043 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.147059 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.147066 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.147139 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.147236 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.147269 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:12:23.148205 5358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.143445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46267 (local address 127.0.98.2:47547)
0928 21:12:23.144152 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.144196 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.145580 (+ 1384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.146087 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.146095 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.146182 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.146796 (+ 614us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.146798 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.146837 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.146903 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.147129 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.147133 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.147137 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.147262 (+ 125us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.147447 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":113,"threads_started":1}
I20260928 21:12:23.149356 5356 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46267
I20260928 21:12:23.149575 5356 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:23.149986 5356 heartbeater.cc:511] Master 127.0.98.62:46267 requested a full tablet report, sending...
I20260928 21:12:23.150801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5228
I20260928 21:12:23.150918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:23.150987 5037 ts_manager.cc:194] Registered new tserver with Master: 358a628e3aea4a10b9f4bbc6cfbb8483 (127.0.98.2:45961)
I20260928 21:12:23.151268 392 external_mini_cluster.cc:1428] Setting key 34456c8caefd589f92cd5c91bc84a91b
I20260928 21:12:23.153149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--builtin_ntp_servers=127.0.98.20:43405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260928 21:12:23.263648 5359 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:23.263886 5359 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:23.263957 5359 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:23.267885 5359 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:23.267941 5359 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:23.268038 5359 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:23.272593 5359 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5359
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:23.273564 5359 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:23.274791 5359 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:23.283890 5367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:23.283921 5365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:23.283949 5364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:23.284358 5359 server_base.cc:1034] running on GCE node
I20260928 21:12:23.284708 5359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:23.285398 5359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:23.286571 5359 hybrid_clock.cc:648] HybridClock initialized: now 1790629943286557 us; error 33 us; skew 500 ppm
I20260928 21:12:23.288458 5359 webserver.cc:460] Webserver started at http://127.0.98.3:42463/ using document root <none> and password file <none>
I20260928 21:12:23.289142 5359 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:23.289233 5359 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:23.289474 5359 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:23.292034 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "4cabd8f4e6fe462f8add56ec698979c7"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "81ad558dc627ab586e999f0287bd133b"
server_key_iv: "8ba84172789a72b90fe2477c87a7f996"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.292711 5359 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4cabd8f4e6fe462f8add56ec698979c7"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "81ad558dc627ab586e999f0287bd133b"
server_key_iv: "8ba84172789a72b90fe2477c87a7f996"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.296381 5359 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:23.298894 5373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.300076 5359 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:23.300197 5359 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "4cabd8f4e6fe462f8add56ec698979c7"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "81ad558dc627ab586e999f0287bd133b"
server_key_iv: "8ba84172789a72b90fe2477c87a7f996"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.300299 5359 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:23.304708 5359 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:23.305285 5359 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:23.305449 5359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:23.305962 5359 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:23.306788 5359 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:23.306833 5359 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.306895 5359 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:23.306928 5359 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.316095 5359 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36953
I20260928 21:12:23.316115 5486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36953 every 8 connection(s)
I20260928 21:12:23.317024 5359 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:23.318945 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5359
I20260928 21:12:23.319067 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:23.319314 392 external_mini_cluster.cc:1428] Setting key ab877fa7ec0d817244b3b528ad973911
I20260928 21:12:23.323289 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.318744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60979 (local address 127.0.98.62:46267)
0928 21:12:23.318909 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.318914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.320129 (+ 1215us) server_negotiation.cc:407] Connection header received
0928 21:12:23.321643 (+ 1514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.321647 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.321687 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.321775 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.322829 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.322832 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.322835 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.322847 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.322867 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.322880 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.322968 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.323118 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.323162 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:12:23.324122 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.319234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46267 (local address 127.0.98.3:60979)
0928 21:12:23.319960 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.320029 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.321435 (+ 1406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.321925 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.321932 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.322029 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.322578 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.322580 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.322620 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.322687 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.322956 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.322964 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.322987 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.323162 (+ 175us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.323414 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":147,"threads_started":1}
I20260928 21:12:23.325168 5487 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46267
I20260928 21:12:23.325397 5487 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:23.325827 5487 heartbeater.cc:511] Master 127.0.98.62:46267 requested a full tablet report, sending...
I20260928 21:12:23.326785 5037 ts_manager.cc:194] Registered new tserver with Master: 4cabd8f4e6fe462f8add56ec698979c7 (127.0.98.3:36953)
I20260928 21:12:23.333421 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:12:23.336928 5076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.335243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42422 (local address 127.0.98.62:46267)
0928 21:12:23.335387 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.335393 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.335566 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:12:23.335707 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.335710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.335748 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.335826 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.336494 (+ 668us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.336498 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.336501 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.336511 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.336530 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.336545 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.336624 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.336732 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.336773 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:12:23.339813 5037 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:23.342947 5037 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:23.354214 5500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.351159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45961 (local address 127.0.0.1:55434)
0928 21:12:23.351677 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.351691 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.351769 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.352550 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.352553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.352576 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.353435 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.353437 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.353446 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.353464 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.353857 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.353861 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.353864 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.353973 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.354014 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":100,"threads_started":1}
I20260928 21:12:23.354357 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.351095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36572 (local address 127.0.98.3:36953)
0928 21:12:23.351874 (+ 779us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.351887 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.351910 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:12:23.351965 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.351983 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.352188 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.352867 (+ 679us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.353595 (+ 728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.353603 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.353616 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.353626 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.353669 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.353687 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.353802 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.354097 (+ 295us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.354175 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":138,"threads_started":1}
I20260928 21:12:23.354221 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.350921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36953 (local address 127.0.0.1:36572)
0928 21:12:23.351291 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.351345 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.351452 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.352363 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.352370 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.352525 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.353214 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.353218 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.353239 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.353314 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.353773 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.353781 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.353787 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.353894 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.353963 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260928 21:12:23.354950 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.351658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55434 (local address 127.0.98.2:45961)
0928 21:12:23.352098 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.352110 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.352134 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:12:23.352199 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.352206 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.352392 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.352834 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.353595 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.353604 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.353616 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.353627 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.353711 (+ 84us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.353741 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.354591 (+ 850us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.354642 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.354711 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":130,"threads_started":1}
I20260928 21:12:23.356055 5501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.351663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35557 (local address 127.0.0.1:32978)
0928 21:12:23.352565 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.352580 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.352671 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.353872 (+ 1201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.353878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.353894 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.354803 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.354808 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.354819 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.354848 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.355802 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.355805 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.355808 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.355894 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.355931 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":85,"threads_started":1}
I20260928 21:12:23.356248 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.352049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32978 (local address 127.0.98.1:35557)
0928 21:12:23.352498 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.352511 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.352678 (+ 167us) server_negotiation.cc:407] Connection header received
0928 21:12:23.353271 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.353281 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.353513 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.353646 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.355605 (+ 1959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.355611 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.355621 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.355632 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.355672 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.355698 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.355827 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.355910 (+ 83us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.355973 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":142,"threads_started":1}
I20260928 21:12:23.357738 5421 tablet_service.cc:1467] Processing CreateTablet for tablet 91a62b3553484c3d8755283be9f7f408 (DEFAULT_TABLE table=test-table [id=19ef331fa4c54d618e7ee507c37d0208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:23.357736 5290 tablet_service.cc:1467] Processing CreateTablet for tablet 91a62b3553484c3d8755283be9f7f408 (DEFAULT_TABLE table=test-table [id=19ef331fa4c54d618e7ee507c37d0208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:23.358884 5421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a62b3553484c3d8755283be9f7f408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:23.358884 5290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a62b3553484c3d8755283be9f7f408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:23.358760 5159 tablet_service.cc:1467] Processing CreateTablet for tablet 91a62b3553484c3d8755283be9f7f408 (DEFAULT_TABLE table=test-table [id=19ef331fa4c54d618e7ee507c37d0208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:23.359615 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91a62b3553484c3d8755283be9f7f408. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:23.365965 5504 tablet_bootstrap.cc:492] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: Bootstrap starting.
I20260928 21:12:23.366923 5506 tablet_bootstrap.cc:492] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Bootstrap starting.
I20260928 21:12:23.368402 5504 tablet_bootstrap.cc:654] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:23.368709 5506 tablet_bootstrap.cc:654] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:23.368887 5505 tablet_bootstrap.cc:492] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Bootstrap starting.
I20260928 21:12:23.369361 5504 log.cc:824] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:23.369517 5506 log.cc:824] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:23.371018 5505 tablet_bootstrap.cc:654] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:23.371869 5504 tablet_bootstrap.cc:492] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: No bootstrap required, opened a new log
I20260928 21:12:23.371869 5506 tablet_bootstrap.cc:492] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: No bootstrap required, opened a new log
I20260928 21:12:23.371848 5505 log.cc:824] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:23.372078 5504 ts_tablet_manager.cc:1397] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:12:23.372078 5506 ts_tablet_manager.cc:1397] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:12:23.373612 5505 tablet_bootstrap.cc:492] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: No bootstrap required, opened a new log
I20260928 21:12:23.373814 5505 ts_tablet_manager.cc:1397] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:12:23.376741 5506 raft_consensus.cc:348] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.376741 5504 raft_consensus.cc:348] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.377063 5504 raft_consensus.cc:374] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:23.377063 5506 raft_consensus.cc:374] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:23.377125 5504 raft_consensus.cc:729] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c5311b75deb4d2ebca9d7779a0d90f4, State: Initialized, Role: FOLLOWER
I20260928 21:12:23.377132 5506 raft_consensus.cc:729] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cabd8f4e6fe462f8add56ec698979c7, State: Initialized, Role: FOLLOWER
I20260928 21:12:23.377656 5506 consensus_queue.cc:260] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.377656 5504 consensus_queue.cc:260] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.377632 5505 raft_consensus.cc:348] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.377866 5505 raft_consensus.cc:374] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:23.377912 5505 raft_consensus.cc:729] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 358a628e3aea4a10b9f4bbc6cfbb8483, State: Initialized, Role: FOLLOWER
I20260928 21:12:23.378391 5505 consensus_queue.cc:260] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.378614 5225 heartbeater.cc:503] Master 127.0.98.62:46267 was elected leader, sending a full tablet report...
I20260928 21:12:23.378720 5504 ts_tablet_manager.cc:1428] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:12:23.378720 5506 ts_tablet_manager.cc:1428] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:12:23.379340 5505 ts_tablet_manager.cc:1428] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:12:23.379683 5356 heartbeater.cc:503] Master 127.0.98.62:46267 was elected leader, sending a full tablet report...
I20260928 21:12:23.379837 5487 heartbeater.cc:503] Master 127.0.98.62:46267 was elected leader, sending a full tablet report...
W20260928 21:12:23.392122 5357 tablet.cc:2369] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:23.468405 5226 tablet.cc:2369] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:23.567410 5511 raft_consensus.cc:482] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:23.567674 5511 raft_consensus.cc:504] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
W20260928 21:12:23.568013 5488 tablet.cc:2369] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:23.568924 5511 leader_election.cc:288] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3c5311b75deb4d2ebca9d7779a0d90f4 (127.0.98.1:35557), 358a628e3aea4a10b9f4bbc6cfbb8483 (127.0.98.2:45961)
I20260928 21:12:23.570727 5489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.569150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45961 (local address 127.0.98.3:49505)
0928 21:12:23.569250 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.569266 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.569357 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.569607 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.569610 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.569626 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.570134 (+ 508us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.570136 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.570149 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.570180 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.570476 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.570479 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.570483 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.570571 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.570610 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:12:23.570760 5502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.569243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49505 (local address 127.0.98.2:45961)
0928 21:12:23.569370 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.569374 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.569421 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:12:23.569461 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.569463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.569493 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.569556 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.570323 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.570330 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.570336 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.570354 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.570378 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.570388 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.570468 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.570570 (+ 102us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.570609 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:12:23.571242 5503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.569243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40583 (local address 127.0.98.1:35557)
0928 21:12:23.569370 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.569375 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.569873 (+ 498us) server_negotiation.cc:407] Connection header received
0928 21:12:23.569964 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.569966 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.570035 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.570136 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.570834 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.570838 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.570840 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.570855 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.570880 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.570890 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.570986 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.571097 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.571134 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:12:23.571256 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.569150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35557 (local address 127.0.98.3:40583)
0928 21:12:23.569756 (+ 606us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.569768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.569885 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.570158 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.570160 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.570172 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:23.570676 (+ 504us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:23.570678 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:23.570687 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:23.570711 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:23.571009 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:23.571012 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:23.571015 (+ 3us) client_negotiation.cc:769] Sending connection context
0928 21:12:23.571089 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:12:23.571124 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"mutex_wait_us":270,"thread_start_us":135,"threads_started":1}
I20260928 21:12:23.571348 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a62b3553484c3d8755283be9f7f408" candidate_uuid: "4cabd8f4e6fe462f8add56ec698979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" is_pre_election: true
I20260928 21:12:23.571659 5310 raft_consensus.cc:2454] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cabd8f4e6fe462f8add56ec698979c7 in term 0.
I20260928 21:12:23.571861 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a62b3553484c3d8755283be9f7f408" candidate_uuid: "4cabd8f4e6fe462f8add56ec698979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" is_pre_election: true
I20260928 21:12:23.572165 5179 raft_consensus.cc:2454] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cabd8f4e6fe462f8add56ec698979c7 in term 0.
I20260928 21:12:23.572202 5377 leader_election.cc:302] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 358a628e3aea4a10b9f4bbc6cfbb8483, 4cabd8f4e6fe462f8add56ec698979c7; no voters:
I20260928 21:12:23.572480 5511 raft_consensus.cc:2781] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:23.572582 5511 raft_consensus.cc:482] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:23.572641 5511 raft_consensus.cc:3037] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:23.573741 5511 raft_consensus.cc:504] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.574091 5511 leader_election.cc:288] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [CANDIDATE]: Term 1 election: Requested vote from peers 3c5311b75deb4d2ebca9d7779a0d90f4 (127.0.98.1:35557), 358a628e3aea4a10b9f4bbc6cfbb8483 (127.0.98.2:45961)
I20260928 21:12:23.574467 5310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a62b3553484c3d8755283be9f7f408" candidate_uuid: "4cabd8f4e6fe462f8add56ec698979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483"
I20260928 21:12:23.574482 5179 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91a62b3553484c3d8755283be9f7f408" candidate_uuid: "4cabd8f4e6fe462f8add56ec698979c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4"
I20260928 21:12:23.574602 5310 raft_consensus.cc:3037] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:23.574601 5179 raft_consensus.cc:3037] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:23.575872 5179 raft_consensus.cc:2454] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cabd8f4e6fe462f8add56ec698979c7 in term 1.
I20260928 21:12:23.575873 5310 raft_consensus.cc:2454] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cabd8f4e6fe462f8add56ec698979c7 in term 1.
I20260928 21:12:23.576170 5375 leader_election.cc:302] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3c5311b75deb4d2ebca9d7779a0d90f4, 4cabd8f4e6fe462f8add56ec698979c7; no voters:
I20260928 21:12:23.576397 5511 raft_consensus.cc:2781] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:23.576639 5511 raft_consensus.cc:686] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 1 LEADER]: Becoming Leader. State: Replica: 4cabd8f4e6fe462f8add56ec698979c7, State: Running, Role: LEADER
I20260928 21:12:23.576998 5511 consensus_queue.cc:237] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } }
I20260928 21:12:23.580247 5035 catalog_manager.cc:5495] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cabd8f4e6fe462f8add56ec698979c7 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4cabd8f4e6fe462f8add56ec698979c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4cabd8f4e6fe462f8add56ec698979c7" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36953 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:23.592402 5499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.590992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49726 (local address 127.0.98.3:36953)
0928 21:12:23.591114 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.591118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.591131 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:23.591266 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.591268 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.591329 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.591400 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.592039 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.592042 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.592044 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.592054 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.592073 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.592081 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.592151 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.592205 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.592237 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:23.598184 5179 raft_consensus.cc:1261] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4 [term 1 FOLLOWER]: Refusing update from remote peer 4cabd8f4e6fe462f8add56ec698979c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:23.598184 5310 raft_consensus.cc:1261] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 1 FOLLOWER]: Refusing update from remote peer 4cabd8f4e6fe462f8add56ec698979c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:23.598874 5514 consensus_queue.cc:1035] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3c5311b75deb4d2ebca9d7779a0d90f4" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:23.599089 5511 consensus_queue.cc:1035] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "358a628e3aea4a10b9f4bbc6cfbb8483" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:23.615329 5036 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:23.615561 5036 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:23.618095 5036 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9c2c613b8ea14a90816a598729663d82: Sending DeleteTablet for 3 replicas of tablet 91a62b3553484c3d8755283be9f7f408
I20260928 21:12:23.618901 5159 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a62b3553484c3d8755283be9f7f408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:23 UTC) from {username='slave'} at 127.0.0.1:32978
I20260928 21:12:23.618927 5290 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a62b3553484c3d8755283be9f7f408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:23 UTC) from {username='slave'} at 127.0.0.1:55434
I20260928 21:12:23.619065 5421 tablet_service.cc:1514] Processing DeleteTablet for tablet 91a62b3553484c3d8755283be9f7f408 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:23 UTC) from {username='slave'} at 127.0.0.1:36572
I20260928 21:12:23.619432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5097
I20260928 21:12:23.619508 5526 tablet_replica.cc:330] T 91a62b3553484c3d8755283be9f7f408 P 3c5311b75deb4d2ebca9d7779a0d90f4: stopping tablet replica
I20260928 21:12:23.620047 5527 tablet_replica.cc:330] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: stopping tablet replica
I20260928 21:12:23.620316 5527 raft_consensus.cc:2229] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:23.620637 5527 raft_consensus.cc:2258] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:23.621024 5528 tablet_replica.cc:330] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: stopping tablet replica
I20260928 21:12:23.621265 5528 raft_consensus.cc:2229] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:23.621642 5528 raft_consensus.cc:2258] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:23.623109 5528 ts_tablet_manager.cc:1905] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:23.623113 5527 ts_tablet_manager.cc:1905] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:23.625922 5528 ts_tablet_manager.cc:1918] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:23.626024 5528 log.cc:1196] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/91a62b3553484c3d8755283be9f7f408
I20260928 21:12:23.626354 5528 ts_tablet_manager.cc:1939] T 91a62b3553484c3d8755283be9f7f408 P 4cabd8f4e6fe462f8add56ec698979c7: Deleting consensus metadata
I20260928 21:12:23.626737 5527 ts_tablet_manager.cc:1918] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:23.626900 5527 log.cc:1196] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/91a62b3553484c3d8755283be9f7f408
I20260928 21:12:23.627347 5527 ts_tablet_manager.cc:1939] T 91a62b3553484c3d8755283be9f7f408 P 358a628e3aea4a10b9f4bbc6cfbb8483: Deleting consensus metadata
I20260928 21:12:23.627588 5022 catalog_manager.cc:4841] TS 4cabd8f4e6fe462f8add56ec698979c7 (127.0.98.3:36953): tablet 91a62b3553484c3d8755283be9f7f408 (table test-table [id=19ef331fa4c54d618e7ee507c37d0208]) successfully deleted
W20260928 21:12:23.627877 5022 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35557 (error 108)
I20260928 21:12:23.628093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5228
I20260928 21:12:23.634218 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5359
I20260928 21:12:23.635036 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.634526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35557 (local address 127.0.0.1:34898)
0928 21:12:23.634876 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.634945 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":275}
I20260928 21:12:23.635251 5498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.634737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45961 (local address 127.0.0.1:58806)
0928 21:12:23.635167 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.635207 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384}
W20260928 21:12:23.635541 5024 catalog_manager.cc:4584] TS 358a628e3aea4a10b9f4bbc6cfbb8483 (127.0.98.2:45961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91a62b3553484c3d8755283be9f7f408: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45961: connect: Connection refused (error 111)
I20260928 21:12:23.640089 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5006
2026-09-28T21:12:23Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1178 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260928 21:12:23.661585 392 mini_oidc.cc:152] Starting JWKS server
I20260928 21:12:23.663285 392 webserver.cc:460] Webserver started at https://127.0.0.1:37605/ using document root <none> and password file <none>
I20260928 21:12:23.663466 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:12:23.663744 392 webserver.cc:460] Webserver started at http://127.0.0.1:42483/ using document root <none> and password file <none>
2026-09-28T21:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:23Z Disabled control of system clock
I20260928 21:12:23.672545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34797
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41647
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34797
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37605/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/testchainca.cert with env {}
W20260928 21:12:23.781232 5535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:23.781493 5535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:23.781533 5535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:23.785434 5535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:23.785480 5535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:23.785495 5535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:23.785511 5535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:23.785526 5535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:12:23.785542 5535 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:12:23.785555 5535 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37605/jwks.json
I20260928 21:12:23.790781 5535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41647
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34797
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37605/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:23.791702 5535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:23.792601 5535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:23.800617 5541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:23.800644 5540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:23.800647 5543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:23.800930 5535 server_base.cc:1034] running on GCE node
I20260928 21:12:23.801434 5535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:23.802466 5535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:23.803625 5535 hybrid_clock.cc:648] HybridClock initialized: now 1790629943803569 us; error 72 us; skew 500 ppm
I20260928 21:12:23.805469 5535 webserver.cc:460] Webserver started at http://127.0.98.62:46113/ using document root <none> and password file <none>
I20260928 21:12:23.806026 5535 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:23.806087 5535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:23.806303 5535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:23.808285 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "238dbe33d9c74689a9db6b7368fbb24b"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "d038504298561c6d931dbf50ec8ac83d"
server_key_iv: "17b07bcc26483de9f0dd75793895a7ff"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.808817 5535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "238dbe33d9c74689a9db6b7368fbb24b"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "d038504298561c6d931dbf50ec8ac83d"
server_key_iv: "17b07bcc26483de9f0dd75793895a7ff"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.812390 5535 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 21:12:23.814793 5549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:23.815959 5535 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:23.816084 5535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "238dbe33d9c74689a9db6b7368fbb24b"
format_stamp: "Formatted at 2026-09-28 21:12:23 on dist-test-slave-v7xz"
server_key: "d038504298561c6d931dbf50ec8ac83d"
server_key_iv: "17b07bcc26483de9f0dd75793895a7ff"
server_key_version: "encryptionkey@0"
I20260928 21:12:23.816185 5535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:23.833287 5535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:23.833981 5535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:23.834168 5535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:23.840969 5535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34797
I20260928 21:12:23.841018 5601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34797 every 8 connection(s)
I20260928 21:12:23.841925 5535 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:23.845055 5602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:23.848039 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5535
I20260928 21:12:23.848176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:23.848430 392 external_mini_cluster.cc:1428] Setting key fa127a68b27c3647b937957ac6a0e217
I20260928 21:12:23.851462 5602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b: Bootstrap starting.
I20260928 21:12:23.853665 5602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:23.854374 5602 log.cc:824] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:23.856137 5602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b: No bootstrap required, opened a new log
I20260928 21:12:23.858052 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.849693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.0.98.62:34797)
0928 21:12:23.850328 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.850344 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.850384 (+ 40us) server_negotiation.cc:407] Connection header received
0928 21:12:23.851068 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.851085 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.851502 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.851973 (+ 471us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:23.852441 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:23.853546 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:23.854550 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:23.855067 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:23.855262 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:23.856167 (+ 905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:23.856173 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:23.856186 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:23.856655 (+ 469us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:23.856714 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:23.856743 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:23.856893 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:23.857012 (+ 119us) server_negotiation.cc:299] Negotiation successful
0928 21:12:23.857295 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":101,"threads_started":1}
I20260928 21:12:23.860045 5602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238dbe33d9c74689a9db6b7368fbb24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34797 } }
I20260928 21:12:23.860311 5602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:23.860383 5602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 238dbe33d9c74689a9db6b7368fbb24b, State: Initialized, Role: FOLLOWER
I20260928 21:12:23.860854 5602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238dbe33d9c74689a9db6b7368fbb24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34797 } }
I20260928 21:12:23.861030 5602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:23.861095 5602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:23.861099 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:12:23.861179 5602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:23.862083 5602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238dbe33d9c74689a9db6b7368fbb24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34797 } }
I20260928 21:12:23.862434 5602 leader_election.cc:302] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 238dbe33d9c74689a9db6b7368fbb24b; no voters:
I20260928 21:12:23.862778 5602 leader_election.cc:288] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:23.863010 5607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:23.863183 5607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [term 1 LEADER]: Becoming Leader. State: Replica: 238dbe33d9c74689a9db6b7368fbb24b, State: Running, Role: LEADER
I20260928 21:12:23.863471 5607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238dbe33d9c74689a9db6b7368fbb24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34797 } }
I20260928 21:12:23.864102 5602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:23.865077 5609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 238dbe33d9c74689a9db6b7368fbb24b. Latest consensus state: current_term: 1 leader_uuid: "238dbe33d9c74689a9db6b7368fbb24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238dbe33d9c74689a9db6b7368fbb24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34797 } } }
I20260928 21:12:23.865196 5609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:23.865584 5616 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:23.865532 5608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "238dbe33d9c74689a9db6b7368fbb24b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "238dbe33d9c74689a9db6b7368fbb24b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34797 } } }
I20260928 21:12:23.865700 5608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:23.868947 5616 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:23.874711 5616 catalog_manager.cc:1347] Generated new cluster ID: 82dea6a73fee406f931fa06bb8310620
I20260928 21:12:23.874776 5616 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:23.891549 5616 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:23.892948 5616 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:23.905956 5616 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 238dbe33d9c74689a9db6b7368fbb24b: Generated new TSK 0
I20260928 21:12:23.906703 5616 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:12:23.939471 5632 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260928 21:12:23.939608 5632 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:12:23.940115 5605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:23.923123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.0.98.62:34797)
0928 21:12:23.923256 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:23.923261 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:23.923447 (+ 186us) server_negotiation.cc:407] Connection header received
0928 21:12:23.923637 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:23.923641 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:23.923695 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:23.923812 (+ 117us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:12:23.924031 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:23.924896 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:23.925758 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:23.926115 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:23.926221 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:23.926528 (+ 307us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:12:23.939693 (+ 13165us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:12:23.939977 (+ 284us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59272: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:12:23.940201 5605 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59272: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260928 21:12:23.941340 5631 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:23.923031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34797 (local address 127.0.0.1:59272)
0928 21:12:23.923362 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:23.923376 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:23.923493 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:23.923829 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:23.923831 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:23.923837 (+ 6us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:12:23.923934 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:23.923939 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:23.924996 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:23.924998 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:23.925632 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:23.925639 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:23.926214 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:23.926230 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:23.926298 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:23.926339 (+ 41us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:12:23.939944 (+ 13605us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0928 21:12:23.940200 (+ 256us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:34797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":115,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:34797: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37605/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260928 21:12:23.942071 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5535
2026-09-28T21:12:23Z chronyd exiting
I20260928 21:12:23.979796 392 test_util.cc:175] -----------------------------------------------
I20260928 21:12:23.979890 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790629916120794-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (319 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260928 21:12:23.980985 392 mini_oidc.cc:152] Starting JWKS server
I20260928 21:12:23.981853 392 webserver.cc:460] Webserver started at https://127.0.0.1:42317/ using document root <none> and password file <none>
I20260928 21:12:23.981958 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:12:23.982237 392 webserver.cc:460] Webserver started at http://127.0.0.1:40765/ using document root <none> and password file <none>
2026-09-28T21:12:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:23Z Disabled control of system clock
I20260928 21:12:23.991225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35881
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37963
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35881
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42317/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/testchainca.cert with env {}
W20260928 21:12:24.106853 5638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:24.107147 5638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:24.107189 5638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:24.111321 5638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:24.111402 5638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:24.111418 5638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:24.111434 5638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:24.111451 5638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:12:24.111467 5638 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:12:24.111481 5638 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42317/jwks.json
I20260928 21:12:24.117111 5638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37963
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35881
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42317/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:24.118175 5638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:24.119129 5638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:24.127744 5638 server_base.cc:1034] running on GCE node
W20260928 21:12:24.127725 5643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:24.127828 5644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:24.127763 5646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:24.128618 5638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:24.129889 5638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:24.131109 5638 hybrid_clock.cc:648] HybridClock initialized: now 1790629944131089 us; error 37 us; skew 500 ppm
I20260928 21:12:24.133232 5638 webserver.cc:460] Webserver started at http://127.0.98.62:33187/ using document root <none> and password file <none>
I20260928 21:12:24.133983 5638 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:24.134092 5638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:24.134348 5638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:24.137586 5638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "60a53e3459a341fbb433fb4da07faa9e"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "e5591e64fc446f915d9849e3b5ca71c8"
server_key_iv: "b1f62cfc8a362ddaece4c0a66f42a515"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.138405 5638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "60a53e3459a341fbb433fb4da07faa9e"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "e5591e64fc446f915d9849e3b5ca71c8"
server_key_iv: "b1f62cfc8a362ddaece4c0a66f42a515"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.143365 5638 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260928 21:12:24.145927 5652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:24.147260 5638 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:24.147392 5638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "60a53e3459a341fbb433fb4da07faa9e"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "e5591e64fc446f915d9849e3b5ca71c8"
server_key_iv: "b1f62cfc8a362ddaece4c0a66f42a515"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.147508 5638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:24.166971 5638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:24.167727 5638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:24.167927 5638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:24.175773 5638 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35881
I20260928 21:12:24.175830 5704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35881 every 8 connection(s)
I20260928 21:12:24.176810 5638 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:24.177490 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5638
I20260928 21:12:24.177621 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:24.177901 392 external_mini_cluster.cc:1428] Setting key cf73344ed66e45bb77b263c99fe05be2
I20260928 21:12:24.180301 5705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:24.186645 5705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e: Bootstrap starting.
I20260928 21:12:24.187258 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:24.179276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38642 (local address 127.0.98.62:35881)
0928 21:12:24.179891 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:24.179907 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:24.179948 (+ 41us) server_negotiation.cc:407] Connection header received
0928 21:12:24.180642 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:24.180659 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:24.181119 (+ 460us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:24.181394 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:24.181727 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.182790 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.183994 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.184465 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.184590 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.185509 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:24.185514 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:24.185526 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:24.185927 (+ 401us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:24.185982 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:24.186002 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:24.186079 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:24.186238 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 21:12:24.186520 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246,"thread_start_us":132,"threads_started":1}
I20260928 21:12:24.188979 5705 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:24.189833 5705 log.cc:824] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:24.190179 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:12:24.191864 5705 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e: No bootstrap required, opened a new log
I20260928 21:12:24.195178 5705 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a53e3459a341fbb433fb4da07faa9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260928 21:12:24.195398 5705 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:24.195457 5705 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60a53e3459a341fbb433fb4da07faa9e, State: Initialized, Role: FOLLOWER
I20260928 21:12:24.195899 5705 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a53e3459a341fbb433fb4da07faa9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260928 21:12:24.196040 5705 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:24.196112 5705 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:24.196197 5705 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:24.197260 5705 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a53e3459a341fbb433fb4da07faa9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260928 21:12:24.197588 5705 leader_election.cc:302] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60a53e3459a341fbb433fb4da07faa9e; no voters:
I20260928 21:12:24.197882 5705 leader_election.cc:288] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:24.198060 5710 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:24.198343 5710 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [term 1 LEADER]: Becoming Leader. State: Replica: 60a53e3459a341fbb433fb4da07faa9e, State: Running, Role: LEADER
I20260928 21:12:24.198689 5710 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a53e3459a341fbb433fb4da07faa9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } }
I20260928 21:12:24.199190 5705 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:24.200543 5712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60a53e3459a341fbb433fb4da07faa9e. Latest consensus state: current_term: 1 leader_uuid: "60a53e3459a341fbb433fb4da07faa9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a53e3459a341fbb433fb4da07faa9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } } }
I20260928 21:12:24.200675 5712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:24.201050 5711 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60a53e3459a341fbb433fb4da07faa9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60a53e3459a341fbb433fb4da07faa9e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35881 } } }
I20260928 21:12:24.201223 5711 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e [sys.catalog]: This master's current role is: LEADER
W20260928 21:12:24.204119 5725 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:12:24.204190 5725 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:12:24.204298 5720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:24.205061 5720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:24.210988 5720 catalog_manager.cc:1347] Generated new cluster ID: 30657c5fdfb24de79d64c6831bfaa329
I20260928 21:12:24.211071 5720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:24.225761 5720 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:24.227182 5720 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:24.235889 5720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60a53e3459a341fbb433fb4da07faa9e: Generated new TSK 0
I20260928 21:12:24.236871 5720 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260928 21:12:24.266474 5735 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260928 21:12:24.266578 5735 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260928 21:12:24.267343 5708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:24.250866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38646 (local address 127.0.98.62:35881)
0928 21:12:24.251002 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:24.251007 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:24.251172 (+ 165us) server_negotiation.cc:407] Connection header received
0928 21:12:24.251404 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:24.251408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:24.251467 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:24.251585 (+ 118us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:12:24.251881 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.252690 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.253606 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.253952 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.254057 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.254317 (+ 260us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:12:24.266900 (+ 12583us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:12:24.267189 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:12:24.267416 5708 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38646: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260928 21:12:24.267484 5734 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:24.250715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35881 (local address 127.0.0.1:38646)
0928 21:12:24.251085 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:24.251098 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:24.251217 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:24.251608 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:24.251610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:24.251618 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:12:24.251748 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:24.251753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.252805 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.252808 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:24.253490 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:24.253499 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.254046 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.254063 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:24.254126 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.254150 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:12:24.267143 (+ 12993us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0928 21:12:24.267298 (+ 155us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35881: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42317/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":139,"threads_started":1}
I20260928 21:12:24.268325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5638
2026-09-28T21:12:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (327 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260928 21:12:24.308077 392 mini_oidc.cc:152] Starting JWKS server
I20260928 21:12:24.309072 392 webserver.cc:460] Webserver started at https://127.0.0.1:34209/ using document root <none> and password file <none>
I20260928 21:12:24.309165 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260928 21:12:24.309397 392 webserver.cc:460] Webserver started at http://127.0.0.1:37151/ using document root <none> and password file <none>
2026-09-28T21:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:24Z Disabled control of system clock
I20260928 21:12:24.319162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37181
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37181
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34209/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:12:24.433163 5741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:24.433459 5741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:24.433499 5741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:24.437709 5741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:24.437777 5741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:24.437798 5741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:24.437816 5741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:24.437831 5741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:12:24.437887 5741 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:12:24.437909 5741 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34209/jwks.json
I20260928 21:12:24.442706 5741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37181
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34209/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:24.443815 5741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:24.445319 5741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:24.453703 5749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:24.453711 5747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:24.453728 5746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:24.454092 5741 server_base.cc:1034] running on GCE node
I20260928 21:12:24.454639 5741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:24.455782 5741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:24.456967 5741 hybrid_clock.cc:648] HybridClock initialized: now 1790629944456945 us; error 37 us; skew 500 ppm
I20260928 21:12:24.458940 5741 webserver.cc:460] Webserver started at http://127.0.98.62:41671/ using document root <none> and password file <none>
I20260928 21:12:24.459545 5741 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:24.459637 5741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:24.459868 5741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:24.461990 5741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "1688901ef06a4cadbcbc898ff37ce63a"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "84d628869f2584c89ac4ab1c5afc3015"
server_key_iv: "e824a158205273dc61f2f0804109152d"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.462548 5741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "1688901ef06a4cadbcbc898ff37ce63a"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "84d628869f2584c89ac4ab1c5afc3015"
server_key_iv: "e824a158205273dc61f2f0804109152d"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.466383 5741 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:24.468812 5755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:24.470431 5741 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:24.470582 5741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "1688901ef06a4cadbcbc898ff37ce63a"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "84d628869f2584c89ac4ab1c5afc3015"
server_key_iv: "e824a158205273dc61f2f0804109152d"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.470734 5741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:24.481911 5741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:24.482589 5741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:24.482775 5741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:24.489651 5741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37181
I20260928 21:12:24.489665 5807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37181 every 8 connection(s)
I20260928 21:12:24.490608 5741 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:24.493966 5808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:24.495549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5741
I20260928 21:12:24.495668 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:24.495954 392 external_mini_cluster.cc:1428] Setting key aefc02acb50faee2b0ee813670d61a3f
I20260928 21:12:24.500674 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a: Bootstrap starting.
I20260928 21:12:24.503134 5808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:24.503945 5808 log.cc:824] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:24.505520 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:24.497519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60624 (local address 127.0.98.62:37181)
0928 21:12:24.498193 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:24.498208 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:24.498245 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:12:24.498974 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:24.498993 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:24.499391 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:24.499738 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:24.500165 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.501168 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.502043 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.502519 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.502676 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.503582 (+ 906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:24.503587 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:24.503600 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:24.503996 (+ 396us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:24.504053 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:24.504094 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:24.504191 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:24.504368 (+ 177us) server_negotiation.cc:299] Negotiation successful
0928 21:12:24.504677 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":138,"threads_started":1}
I20260928 21:12:24.506306 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a: No bootstrap required, opened a new log
I20260928 21:12:24.508458 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:12:24.511415 5808 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1688901ef06a4cadbcbc898ff37ce63a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260928 21:12:24.511672 5808 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:24.511767 5808 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1688901ef06a4cadbcbc898ff37ce63a, State: Initialized, Role: FOLLOWER
I20260928 21:12:24.512254 5808 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1688901ef06a4cadbcbc898ff37ce63a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260928 21:12:24.512404 5808 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:24.512470 5808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:24.512566 5808 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:24.513602 5808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1688901ef06a4cadbcbc898ff37ce63a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260928 21:12:24.513974 5808 leader_election.cc:302] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1688901ef06a4cadbcbc898ff37ce63a; no voters:
I20260928 21:12:24.514305 5808 leader_election.cc:288] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:24.514427 5813 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:24.514683 5813 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [term 1 LEADER]: Becoming Leader. State: Replica: 1688901ef06a4cadbcbc898ff37ce63a, State: Running, Role: LEADER
I20260928 21:12:24.514974 5813 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1688901ef06a4cadbcbc898ff37ce63a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } }
I20260928 21:12:24.515596 5808 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:24.516523 5814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1688901ef06a4cadbcbc898ff37ce63a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1688901ef06a4cadbcbc898ff37ce63a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } } }
I20260928 21:12:24.516650 5814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:24.517045 5822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:24.518451 5815 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1688901ef06a4cadbcbc898ff37ce63a. Latest consensus state: current_term: 1 leader_uuid: "1688901ef06a4cadbcbc898ff37ce63a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1688901ef06a4cadbcbc898ff37ce63a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37181 } } }
I20260928 21:12:24.518556 5815 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:24.520021 5822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:24.525441 5822 catalog_manager.cc:1347] Generated new cluster ID: 7336e65c3c8848a2a4d3de1ee0f2c39e
I20260928 21:12:24.525542 5822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:24.536659 5822 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:24.537804 5822 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:24.545392 5822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1688901ef06a4cadbcbc898ff37ce63a: Generated new TSK 0
I20260928 21:12:24.546176 5822 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:24.559311 5811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:24.554146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60636 (local address 127.0.98.62:37181)
0928 21:12:24.554288 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:24.554293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:24.554508 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:12:24.554670 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:24.554672 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:24.554721 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:24.554827 (+ 106us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:12:24.555094 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.555621 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.557059 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.557279 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.557371 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.557693 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:12:24.558907 (+ 1214us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:12:24.559164 (+ 257us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 21:12:24.559382 5837 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:24.554060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37181 (local address 127.0.0.1:60636)
0928 21:12:24.554398 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:24.554416 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:24.554544 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:24.554851 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:24.554855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:24.554867 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:12:24.554991 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:24.554998 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.555751 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.555755 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:24.556941 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:24.556952 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.557378 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.557404 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:24.557502 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.557533 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:12:24.559069 (+ 1536us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0928 21:12:24.559215 (+ 146us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37181: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":109,"threads_started":1}
W20260928 21:12:24.559396 5811 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60636: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34209/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
I20260928 21:12:24.560158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5741
E20260928 21:12:24.571799 5838 webserver.cc:533] Webserver: sslize failed:
E20260928 21:12:24.571873 5838 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-28T21:12:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (291 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-28T21:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:24Z Disabled control of system clock
I20260928 21:12:24.608210 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38951
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38951
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/testchainca.cert with env {}
W20260928 21:12:24.718773 5842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:24.719071 5842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:24.719111 5842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:24.723155 5842 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:24.723227 5842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:24.723243 5842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:24.723260 5842 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:24.723276 5842 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:12:24.723299 5842 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260928 21:12:24.723315 5842 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260928 21:12:24.728875 5842 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38951
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5842
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:24.729964 5842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:24.730970 5842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:24.739857 5848 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:24.739948 5847 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:24.739857 5850 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:24.740404 5842 server_base.cc:1034] running on GCE node
I20260928 21:12:24.741014 5842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:24.742110 5842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:24.743250 5842 hybrid_clock.cc:648] HybridClock initialized: now 1790629944743235 us; error 31 us; skew 500 ppm
I20260928 21:12:24.745340 5842 webserver.cc:460] Webserver started at http://127.0.98.62:42651/ using document root <none> and password file <none>
I20260928 21:12:24.745955 5842 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:24.746068 5842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:24.746289 5842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:24.748611 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "e09427255b2c47f8b8d1451abb49e9c8"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "4e178209a9f90e00f46eedda098b1bb6"
server_key_iv: "cfb2ae103e40abce34160186443a0504"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.749348 5842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "e09427255b2c47f8b8d1451abb49e9c8"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "4e178209a9f90e00f46eedda098b1bb6"
server_key_iv: "cfb2ae103e40abce34160186443a0504"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.753598 5842 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:24.756247 5856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:24.757706 5842 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:24.757901 5842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "e09427255b2c47f8b8d1451abb49e9c8"
format_stamp: "Formatted at 2026-09-28 21:12:24 on dist-test-slave-v7xz"
server_key: "4e178209a9f90e00f46eedda098b1bb6"
server_key_iv: "cfb2ae103e40abce34160186443a0504"
server_key_version: "encryptionkey@0"
I20260928 21:12:24.758112 5842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:24.771883 5842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:24.772601 5842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:24.772801 5842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:24.779587 5842 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38951
I20260928 21:12:24.779598 5908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38951 every 8 connection(s)
I20260928 21:12:24.780557 5842 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:24.783668 5909 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:24.784008 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5842
I20260928 21:12:24.784130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:24.784440 392 external_mini_cluster.cc:1428] Setting key 643da82383d3242ade44c7f023a1319c
I20260928 21:12:24.790217 5909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8: Bootstrap starting.
I20260928 21:12:24.792358 5909 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:24.793200 5909 log.cc:824] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:24.793485 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:24.785803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51278 (local address 127.0.98.62:38951)
0928 21:12:24.786416 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:24.786431 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:24.786464 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:12:24.787131 (+ 667us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:24.787149 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:24.787533 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:24.787871 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:24.788306 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.789323 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.790190 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.790700 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.790873 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.791739 (+ 866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:24.791744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:24.791756 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:24.792145 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:24.792199 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:24.792219 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:24.792327 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:24.792441 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 21:12:24.792695 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":117,"threads_started":1}
I20260928 21:12:24.795205 5909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8: No bootstrap required, opened a new log
I20260928 21:12:24.796300 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:12:24.798985 5909 raft_consensus.cc:348] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09427255b2c47f8b8d1451abb49e9c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260928 21:12:24.799197 5909 raft_consensus.cc:374] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:24.799270 5909 raft_consensus.cc:729] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e09427255b2c47f8b8d1451abb49e9c8, State: Initialized, Role: FOLLOWER
I20260928 21:12:24.799705 5909 consensus_queue.cc:260] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09427255b2c47f8b8d1451abb49e9c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260928 21:12:24.799845 5909 raft_consensus.cc:388] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:24.799927 5909 raft_consensus.cc:482] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:24.800038 5909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:24.800992 5909 raft_consensus.cc:504] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09427255b2c47f8b8d1451abb49e9c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260928 21:12:24.801347 5909 leader_election.cc:302] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e09427255b2c47f8b8d1451abb49e9c8; no voters:
I20260928 21:12:24.801723 5909 leader_election.cc:288] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:24.801806 5914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:24.802039 5914 raft_consensus.cc:686] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [term 1 LEADER]: Becoming Leader. State: Replica: e09427255b2c47f8b8d1451abb49e9c8, State: Running, Role: LEADER
I20260928 21:12:24.802349 5914 consensus_queue.cc:237] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09427255b2c47f8b8d1451abb49e9c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } }
I20260928 21:12:24.802945 5909 sys_catalog.cc:564] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:24.804028 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e09427255b2c47f8b8d1451abb49e9c8. Latest consensus state: current_term: 1 leader_uuid: "e09427255b2c47f8b8d1451abb49e9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09427255b2c47f8b8d1451abb49e9c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } } }
I20260928 21:12:24.804169 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:24.804427 5915 sys_catalog.cc:455] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e09427255b2c47f8b8d1451abb49e9c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e09427255b2c47f8b8d1451abb49e9c8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38951 } } }
I20260928 21:12:24.804515 5915 sys_catalog.cc:458] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:24.804807 5923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:24.807978 5923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:24.813890 5923 catalog_manager.cc:1347] Generated new cluster ID: eb942fe8919c4ff68b5784df741c503c
I20260928 21:12:24.813968 5923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:24.832860 5923 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:24.834136 5923 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:24.839046 5923 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e09427255b2c47f8b8d1451abb49e9c8: Generated new TSK 0
I20260928 21:12:24.839850 5923 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:24.887985 5912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:24.851700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51294 (local address 127.0.98.62:38951)
0928 21:12:24.851837 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:24.851842 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:24.852021 (+ 179us) server_negotiation.cc:407] Connection header received
0928 21:12:24.852216 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:24.852219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:24.852267 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:24.852380 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0928 21:12:24.852605 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.853210 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.854066 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.854288 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.854395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.854694 (+ 299us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0928 21:12:24.887466 (+ 32772us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:12:24.887814 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51294: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 21:12:24.888088 5938 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:24.851644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38951 (local address 127.0.0.1:51294)
0928 21:12:24.851930 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:24.851943 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:24.852074 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:24.852377 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:24.852380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:24.852388 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0928 21:12:24.852495 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:24.852501 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.853321 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.853325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:24.853940 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:24.853947 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:24.854392 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:24.854412 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:24.854502 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:24.854529 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0928 21:12:24.887710 (+ 33181us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0928 21:12:24.887895 (+ 185us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38951: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":112,"threads_started":1}
W20260928 21:12:24.888088 5912 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51294: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260928 21:12:24.888818 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5842
2026-09-28T21:12:24Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (308 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (126 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:25 dist-test-slave-v7xz krb5kdc[5950](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:25 dist-test-slave-v7xz krb5kdc[5950](info): set up 2 sockets
Sep 28 21:12:25 dist-test-slave-v7xz krb5kdc[5950](info): commencing operation
krb5kdc: starting...
W20260928 21:12:26.193682 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:26 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:26.319319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34875
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34875
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:26.430500 5963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:26.430788 5963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:26.430830 5963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:26.434810 5963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:26.434866 5963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:26.434885 5963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:26.434902 5963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:26.434916 5963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:26.440091 5963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34875
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34875
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:26.441119 5963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:26.442106 5963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:26.450631 5969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:26.450636 5968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:26.450647 5971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:26.450738 5963 server_base.cc:1034] running on GCE node
I20260928 21:12:26.451368 5963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:26.452502 5963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:26.453691 5963 hybrid_clock.cc:648] HybridClock initialized: now 1790629946453577 us; error 129 us; skew 500 ppm
Sep 28 21:12:26 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:26.456498 5963 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:26.457463 5963 webserver.cc:460] Webserver started at http://127.0.98.62:37875/ using document root <none> and password file <none>
I20260928 21:12:26.457998 5963 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:26.458060 5963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:26.458225 5963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:26.460237 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "3c7c3408da2f4660b8354d2c77f8fccc"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "dd168859c8f77d75ea3988c43f5a2777"
server_key_iv: "b228479ed2251eb4ded11d5292d98064"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.460753 5963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c7c3408da2f4660b8354d2c77f8fccc"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "dd168859c8f77d75ea3988c43f5a2777"
server_key_iv: "b228479ed2251eb4ded11d5292d98064"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.464521 5963 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:26.467114 5978 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:26.468333 5963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:26.468493 5963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "3c7c3408da2f4660b8354d2c77f8fccc"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "dd168859c8f77d75ea3988c43f5a2777"
server_key_iv: "b228479ed2251eb4ded11d5292d98064"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.468653 5963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:26.480587 5963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:26.483999 5963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:26.484217 5963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:26.491195 5963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34875
I20260928 21:12:26.491503 6030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34875 every 8 connection(s)
I20260928 21:12:26.492228 5963 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:26.495057 6031 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:26.495404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 5963
I20260928 21:12:26.495533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:26.495826 392 external_mini_cluster.cc:1428] Setting key f73ca273e2dd575fc013a2ee15700d5d
I20260928 21:12:26.502094 6031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc: Bootstrap starting.
I20260928 21:12:26.504995 6031 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc: Neither blocks nor log segments found. Creating new log.
Sep 28 21:12:26 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:26.505836 6031 log.cc:824] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:26.508431 6031 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc: No bootstrap required, opened a new log
I20260928 21:12:26.511180 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:26.497217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.0.98.62:34875)
0928 21:12:26.497631 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:26.497647 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:26.497684 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:12:26.498371 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:26.498388 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:26.498793 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:26.499097 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:26.500008 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:26.501088 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:26.502240 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:26.502737 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:26.502863 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:26.505710 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:26.505717 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:26.505735 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:26.505779 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:26.507806 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:26.508276 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:26.508281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:26.508288 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:26.508449 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:26.508751 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:26.508754 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:26.508757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:26.509305 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:26.509501 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:26.509748 (+ 247us) server_negotiation.cc:299] Negotiation successful
0928 21:12:26.510059 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":142,"threads_started":1}
I20260928 21:12:26.513113 6031 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260928 21:12:26.513372 6031 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:26.513441 6031 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c7c3408da2f4660b8354d2c77f8fccc, State: Initialized, Role: FOLLOWER
I20260928 21:12:26.513877 6031 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260928 21:12:26.514005 6031 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:26.514053 6031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:26.514123 6031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:26.514999 6031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260928 21:12:26.515285 6031 leader_election.cc:302] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c7c3408da2f4660b8354d2c77f8fccc; no voters:
I20260928 21:12:26.515544 6031 leader_election.cc:288] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:26.515838 6036 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:26.516053 6036 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [term 1 LEADER]: Becoming Leader. State: Replica: 3c7c3408da2f4660b8354d2c77f8fccc, State: Running, Role: LEADER
I20260928 21:12:26.516391 6036 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } }
I20260928 21:12:26.516939 6031 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:26.518072 6037 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } } }
I20260928 21:12:26.518208 6037 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:26.518525 6038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c7c3408da2f4660b8354d2c77f8fccc. Latest consensus state: current_term: 1 leader_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c7c3408da2f4660b8354d2c77f8fccc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34875 } } }
I20260928 21:12:26.518644 6038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:26.518585 6045 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:26.521855 6045 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:26.527263 6045 catalog_manager.cc:1347] Generated new cluster ID: 878e5904017e4bd0a0033230a104d381
I20260928 21:12:26.527343 6045 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:26.548959 6045 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:26.549913 6045 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:26.556268 6045 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c7c3408da2f4660b8354d2c77f8fccc: Generated new TSK 0
I20260928 21:12:26.556952 6045 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:26.600252 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:26.714995 6057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:26.715283 6057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:26.715327 6057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:26.719810 6057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:26.719889 6057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:26.719995 6057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:26.725526 6057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:26.726742 6057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:26.727815 6057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:26.737995 6063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:26.738059 6057 server_base.cc:1034] running on GCE node
W20260928 21:12:26.738062 6062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:26.738091 6065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:26.738787 6057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:26.739454 6057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:26.740656 6057 hybrid_clock.cc:648] HybridClock initialized: now 1790629946740634 us; error 38 us; skew 500 ppm
Sep 28 21:12:26 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:26.743535 6057 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:26.744478 6057 webserver.cc:460] Webserver started at http://127.0.98.1:37433/ using document root <none> and password file <none>
I20260928 21:12:26.745132 6057 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:26.745225 6057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:26.745457 6057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:26.747599 6057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "b4ecffb7e77549759817e1e0352e7412"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "cebe0051bc48ee2375475131337bf3b5"
server_key_iv: "cf56bfd8f725a851c882257120e5f092"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.748165 6057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b4ecffb7e77549759817e1e0352e7412"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "cebe0051bc48ee2375475131337bf3b5"
server_key_iv: "cf56bfd8f725a851c882257120e5f092"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.752061 6057 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:26.754774 6072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:26.756130 6057 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:26.756276 6057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "b4ecffb7e77549759817e1e0352e7412"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "cebe0051bc48ee2375475131337bf3b5"
server_key_iv: "cf56bfd8f725a851c882257120e5f092"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.756395 6057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:26.773324 6057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:26.776603 6057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:26.776804 6057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:26.777467 6057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:26.778296 6057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:26.778365 6057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:26.778431 6057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:26.778478 6057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:26.787606 6057 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46501
I20260928 21:12:26.787678 6185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46501 every 8 connection(s)
I20260928 21:12:26.788558 6057 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:26.797003 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6057
I20260928 21:12:26.797145 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:26.797494 392 external_mini_cluster.cc:1428] Setting key e4942a7b9662c4095f6d7b1b1951d99f
Sep 28 21:12:26 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:26.803588 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:26.790440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46347 (local address 127.0.98.62:34875)
0928 21:12:26.790575 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:26.790579 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:26.791502 (+ 923us) server_negotiation.cc:407] Connection header received
0928 21:12:26.792935 (+ 1433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:26.792938 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:26.792983 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:26.793068 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:26.794435 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:26.794960 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:26.796123 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:26.796344 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:26.796422 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:26.799429 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:26.799432 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:26.799434 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:26.799458 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:26.801301 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:26.801864 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:26.801870 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:26.801875 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:26.801968 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:26.802298 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:26.802304 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:26.802309 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:26.802573 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:26.802678 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:26.803194 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 21:12:26.803358 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:12:26.804373 6188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:26.790857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34875 (local address 127.0.98.1:46347)
0928 21:12:26.791344 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:26.791389 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:26.792678 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:26.793264 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:26.793272 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:26.793668 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:26.794273 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:26.794288 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:26.795092 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:26.795096 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:26.796012 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:26.796021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:26.796530 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:26.796534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:26.796704 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:26.797416 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:26.797440 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:26.799214 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:26.801448 (+ 2234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:26.801453 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:26.801467 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:26.801761 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:26.802062 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:26.802064 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:26.802065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:26.802201 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:26.802674 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:26.802680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:26.802929 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 21:12:26.803105 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 21:12:26.803452 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":104,"threads_started":1}
I20260928 21:12:26.805737 6186 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34875
I20260928 21:12:26.806003 6186 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:26.806497 6186 heartbeater.cc:511] Master 127.0.98.62:34875 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:12:26.808101 5995 ts_manager.cc:194] Registered new tserver with Master: b4ecffb7e77549759817e1e0352e7412 (127.0.98.1:46501)
I20260928 21:12:26.809361 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46347
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:26.827781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:26.939563 6191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:26.939863 6191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:26.939908 6191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:26.943974 6191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:26.944046 6191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:26.944109 6191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:26.949601 6191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:26.950712 6191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:26.951715 6191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:26.961366 6199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:26.961329 6196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:26.961298 6197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:26.961709 6191 server_base.cc:1034] running on GCE node
I20260928 21:12:26.962152 6191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:26.962877 6191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:26.964047 6191 hybrid_clock.cc:648] HybridClock initialized: now 1790629946964031 us; error 30 us; skew 500 ppm
Sep 28 21:12:26 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:26.970552 6191 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:26.971524 6191 webserver.cc:460] Webserver started at http://127.0.98.2:35509/ using document root <none> and password file <none>
I20260928 21:12:26.972172 6191 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:26.972235 6191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:26.972452 6191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:26.974843 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "40e2c979af774bd99e7da00c20236bf9"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "b47270bd9974c143141417af08f6d970"
server_key_iv: "764ab230f7441e1984cbbb7c68b9de45"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.975456 6191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "40e2c979af774bd99e7da00c20236bf9"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "b47270bd9974c143141417af08f6d970"
server_key_iv: "764ab230f7441e1984cbbb7c68b9de45"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.979475 6191 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260928 21:12:26.982162 6206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:26.983458 6191 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:26.983603 6191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "40e2c979af774bd99e7da00c20236bf9"
format_stamp: "Formatted at 2026-09-28 21:12:26 on dist-test-slave-v7xz"
server_key: "b47270bd9974c143141417af08f6d970"
server_key_iv: "764ab230f7441e1984cbbb7c68b9de45"
server_key_version: "encryptionkey@0"
I20260928 21:12:26.983743 6191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:27.003032 6191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:27.006572 6191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:27.006795 6191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:27.007521 6191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:27.008397 6191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:27.008445 6191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:27.008486 6191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:27.008539 6191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:27.017799 6191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46439
I20260928 21:12:27.017838 6319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46439 every 8 connection(s)
I20260928 21:12:27.018847 6191 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:27.024565 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6191
I20260928 21:12:27.024706 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:27.025082 392 external_mini_cluster.cc:1428] Setting key 9e585a97b35eeb693e3e3d8522dcf35a
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:27.034842 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.020693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38263 (local address 127.0.98.62:34875)
0928 21:12:27.020957 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.020961 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.021856 (+ 895us) server_negotiation.cc:407] Connection header received
0928 21:12:27.023425 (+ 1569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.023428 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.023487 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.023598 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.024947 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.025662 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.027019 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.027455 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.027580 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.030262 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:27.030269 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:27.030276 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:27.030310 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:27.032323 (+ 2013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.032942 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.032947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.032953 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.033068 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.033470 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.033475 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.033480 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.033740 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:27.033860 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:27.034482 (+ 622us) server_negotiation.cc:299] Negotiation successful
0928 21:12:27.034629 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:27.035763 6322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.021177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34875 (local address 127.0.98.2:38263)
0928 21:12:27.021663 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:27.021710 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:27.023183 (+ 1473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:27.023790 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:27.023799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:27.024187 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:27.024751 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:27.024774 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.025813 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.025817 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:27.026892 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:27.026902 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.027615 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.027618 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:27.027773 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.028450 (+ 677us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:27.028467 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:27.030082 (+ 1615us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:27.032472 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:27.032478 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:27.032493 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:27.032789 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:27.033186 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:27.033189 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:27.033190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:27.033352 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:27.033871 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:27.033877 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:27.034181 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 21:12:27.034376 (+ 195us) client_negotiation.cc:240] Negotiation successful
0928 21:12:27.034698 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":114,"threads_started":1}
I20260928 21:12:27.037278 6320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34875
I20260928 21:12:27.037631 6320 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:27.038259 6320 heartbeater.cc:511] Master 127.0.98.62:34875 requested a full tablet report, sending...
I20260928 21:12:27.039454 5995 ts_manager.cc:194] Registered new tserver with Master: 40e2c979af774bd99e7da00c20236bf9 (127.0.98.2:46439)
I20260928 21:12:27.040159 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:38263
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:27.055585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:27.168779 6325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:27.169190 6325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:27.169267 6325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:27.173636 6325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:27.173713 6325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:27.173802 6325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:27.179126 6325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34875
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:27.180183 6325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:27.181257 6325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:27.190690 6330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:27.190714 6331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:27.190642 6333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:27.191008 6325 server_base.cc:1034] running on GCE node
I20260928 21:12:27.191442 6325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:27.192082 6325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:27.193256 6325 hybrid_clock.cc:648] HybridClock initialized: now 1790629947193224 us; error 52 us; skew 500 ppm
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.195945 6325 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:27.196918 6325 webserver.cc:460] Webserver started at http://127.0.98.3:33693/ using document root <none> and password file <none>
I20260928 21:12:27.197474 6325 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:27.197561 6325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:27.197777 6325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:27.199757 6325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "d65f7dbbe0bf45279fe04b5600bcbca9"
format_stamp: "Formatted at 2026-09-28 21:12:27 on dist-test-slave-v7xz"
server_key: "0b25bddf2c81959a35f3987a0a1bf8cb"
server_key_iv: "47e46e9c4272218ac9373c552d3c5914"
server_key_version: "encryptionkey@0"
I20260928 21:12:27.200286 6325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d65f7dbbe0bf45279fe04b5600bcbca9"
format_stamp: "Formatted at 2026-09-28 21:12:27 on dist-test-slave-v7xz"
server_key: "0b25bddf2c81959a35f3987a0a1bf8cb"
server_key_iv: "47e46e9c4272218ac9373c552d3c5914"
server_key_version: "encryptionkey@0"
I20260928 21:12:27.203900 6325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:12:27.206216 6340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:27.207335 6325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:27.207458 6325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "d65f7dbbe0bf45279fe04b5600bcbca9"
format_stamp: "Formatted at 2026-09-28 21:12:27 on dist-test-slave-v7xz"
server_key: "0b25bddf2c81959a35f3987a0a1bf8cb"
server_key_iv: "47e46e9c4272218ac9373c552d3c5914"
server_key_version: "encryptionkey@0"
I20260928 21:12:27.207558 6325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:27.221850 6325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:27.224666 6325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:27.224900 6325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:27.225476 6325 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:27.226271 6325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:27.226337 6325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:27.226418 6325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:27.226462 6325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:27.235128 6325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45091
I20260928 21:12:27.235157 6453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45091 every 8 connection(s)
I20260928 21:12:27.235999 6325 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:27.241906 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6325
I20260928 21:12:27.242007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:27.242286 392 external_mini_cluster.cc:1428] Setting key 210f97f506abbfb01fd9b2502031d2e1
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:27.249475 6034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.237828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38419 (local address 127.0.98.62:34875)
0928 21:12:27.237964 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.237967 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.238798 (+ 831us) server_negotiation.cc:407] Connection header received
0928 21:12:27.240189 (+ 1391us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.240192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.240248 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.240329 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.241505 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.242108 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.243435 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.243669 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.243775 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.246248 (+ 2473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:27.246251 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:27.246254 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:27.246279 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:27.247543 (+ 1264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.248100 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.248103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.248104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.248188 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.248487 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.248489 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.248490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.248678 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:27.248754 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:27.249256 (+ 502us) server_negotiation.cc:299] Negotiation successful
0928 21:12:27.249359 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:12:27.250329 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.238222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34875 (local address 127.0.98.3:38419)
0928 21:12:27.238642 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:27.238683 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:27.239978 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:27.240499 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:27.240506 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:27.240885 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:27.241359 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:27.241372 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.242243 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.242246 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:27.243325 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:27.243334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.243807 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.243811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:27.243980 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.244571 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:27.244586 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:27.246094 (+ 1508us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:27.247655 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:27.247660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:27.247676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:27.248000 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:27.248273 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:27.248275 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:27.248277 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:27.248411 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:27.248760 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:27.248765 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:27.249053 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 21:12:27.249207 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 21:12:27.249479 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":100,"threads_started":1}
I20260928 21:12:27.251400 6454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34875
I20260928 21:12:27.251636 6454 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:27.252128 6454 heartbeater.cc:511] Master 127.0.98.62:34875 requested a full tablet report, sending...
I20260928 21:12:27.253123 5995 ts_manager.cc:194] Registered new tserver with Master: d65f7dbbe0bf45279fe04b5600bcbca9 (127.0.98.3:45091)
I20260928 21:12:27.253692 5995 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:38419
I20260928 21:12:27.256207 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/test-admin.keytab.
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.273521 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260928 21:12:27.276340 6461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:27.274968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49458 (local address 127.0.98.1:46501)
0928 21:12:27.275284 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.275298 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.275327 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:12:27.275411 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.275416 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.275608 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.275727 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.276164 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49458: BlockingRecv error: recv got EOF from 127.0.0.1:49458 (error 108)
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":114,"threads_started":1}
W20260928 21:12:27.276379 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.277716 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.278615 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:27.289247 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.279439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49464 (local address 127.0.98.1:46501)
0928 21:12:27.280724 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.280727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.280740 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:27.280779 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.280781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.280817 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.280951 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.281495 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.282269 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.283069 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.283449 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.283532 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.285769 (+ 2237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:27.285787 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:27.285833 (+ 46us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:27.285857 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:27.287452 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.287891 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.287896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.287897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.287982 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.288212 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.288214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.288215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.288487 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:27.288643 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:27.288915 (+ 272us) server_negotiation.cc:299] Negotiation successful
0928 21:12:27.289090 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1214}
W20260928 21:12:27.293017 6461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:27.292123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49472 (local address 127.0.98.1:46501)
0928 21:12:27.292365 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.292368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.292379 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:12:27.292409 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.292410 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.292443 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.292506 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.292911 (+ 405us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49472: BlockingRecv error: recv got EOF from 127.0.0.1:49472 (error 108)
Metrics: {"server-negotiator.queue_time_us":153}
W20260928 21:12:27.293104 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.294121 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.294961 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:27.303449 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.295743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49482 (local address 127.0.98.1:46501)
0928 21:12:27.295911 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.295914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.295926 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:27.296000 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.296002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.296039 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.296105 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.296546 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.297149 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.297904 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.298124 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.298225 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.300380 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:27.300384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:27.300386 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:27.300407 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:27.301867 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.302242 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.302245 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.302246 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.302317 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.302557 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.302559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.302560 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.302706 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:27.302798 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:27.303028 (+ 230us) server_negotiation.cc:299] Negotiation successful
0928 21:12:27.303190 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260928 21:12:27.307423 6461 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:27.306516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49498 (local address 127.0.98.1:46501)
0928 21:12:27.306627 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.306632 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.306646 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:27.306681 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.306683 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.306724 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.306810 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.307297 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49498: BlockingRecv error: recv got EOF from 127.0.0.1:49498 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260928 21:12:27.307458 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790629916120794-392-0/krb5kdc/krb5cc' of type FILE
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.308542 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:27.309589 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[5950](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629947, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:12:27.318888 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:27.310572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49506 (local address 127.0.98.1:46501)
0928 21:12:27.310704 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:27.310707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:27.310720 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:27.310809 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:27.310812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:27.310850 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:27.311300 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:27.311795 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.312347 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.313208 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:27.313414 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:27.313488 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:27.315638 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:27.315641 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:27.315643 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:27.315667 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:27.317142 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.317561 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.317564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.317565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.317669 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:27.317973 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:27.317976 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:27.317977 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:27.318138 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:27.318264 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:27.318510 (+ 246us) server_negotiation.cc:299] Negotiation successful
0928 21:12:27.318647 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:12:27.320006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6057
I20260928 21:12:27.326233 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6191
I20260928 21:12:27.332000 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6325
I20260928 21:12:27.338973 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 5963
2026-09-28T21:12:27Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2200 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[6474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[6474](info): set up 2 sockets
Sep 28 21:12:27 dist-test-slave-v7xz krb5kdc[6474](info): commencing operation
krb5kdc: starting...
W20260928 21:12:28.369304 392 mini_kdc.cc:121] Time spent starting KDC: real 1.007s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:28 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 21:12:28.491611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35485
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:28.604895 6487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:28.605233 6487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:28.605307 6487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:28.609413 6487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:28.609512 6487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:28.609548 6487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:28.609587 6487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:28.609640 6487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:28.615042 6487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:28.616237 6487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:28.617350 6487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:28.625753 6495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:28.625761 6493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:28.625842 6492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:28.626044 6487 server_base.cc:1034] running on GCE node
I20260928 21:12:28.626660 6487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:28.627864 6487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:28.629050 6487 hybrid_clock.cc:648] HybridClock initialized: now 1790629948629024 us; error 38 us; skew 500 ppm
Sep 28 21:12:28 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:28.631822 6487 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 21:12:28.632800 6487 webserver.cc:460] Webserver started at http://127.0.98.62:39043/ using document root <none> and password file <none>
I20260928 21:12:28.633414 6487 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:28.633477 6487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:28.633665 6487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:28.635785 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "4061bb912b55432b8bf8cb174d023b32"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "40433fda5c6c361a9c45fa89cd839adf"
server_key_iv: "9df19dcda2bcde2d4e24ecea585e5abd"
server_key_version: "encryptionkey@0"
I20260928 21:12:28.636343 6487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "4061bb912b55432b8bf8cb174d023b32"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "40433fda5c6c361a9c45fa89cd839adf"
server_key_iv: "9df19dcda2bcde2d4e24ecea585e5abd"
server_key_version: "encryptionkey@0"
I20260928 21:12:28.640256 6487 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:28.643086 6502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:28.644551 6487 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:28.644678 6487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "4061bb912b55432b8bf8cb174d023b32"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "40433fda5c6c361a9c45fa89cd839adf"
server_key_iv: "9df19dcda2bcde2d4e24ecea585e5abd"
server_key_version: "encryptionkey@0"
I20260928 21:12:28.644788 6487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:28.657208 6487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:28.660518 6487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:28.660683 6487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:28.668282 6487 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35485
I20260928 21:12:28.668589 6554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35485 every 8 connection(s)
I20260928 21:12:28.669394 6487 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:28.672274 6555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:28.677865 6555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Bootstrap starting.
I20260928 21:12:28.678092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6487
I20260928 21:12:28.678220 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:28.678522 392 external_mini_cluster.cc:1428] Setting key 6a6915f076461c30b66fd0a3e7a9b0f5
I20260928 21:12:28.680686 6555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:28.681470 6555 log.cc:824] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:28.683300 6555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: No bootstrap required, opened a new log
I20260928 21:12:28.686904 6555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260928 21:12:28.687150 6555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:28.687227 6555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4061bb912b55432b8bf8cb174d023b32, State: Initialized, Role: FOLLOWER
Sep 28 21:12:28 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:28.687762 6555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260928 21:12:28.687952 6555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:28.688024 6555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:28.688122 6555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:28.689374 6555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260928 21:12:28.689718 6555 leader_election.cc:302] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4061bb912b55432b8bf8cb174d023b32; no voters:
I20260928 21:12:28.690048 6555 leader_election.cc:288] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:28.690140 6560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:28.690356 6560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 LEADER]: Becoming Leader. State: Replica: 4061bb912b55432b8bf8cb174d023b32, State: Running, Role: LEADER
I20260928 21:12:28.690699 6560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } }
I20260928 21:12:28.691381 6555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:28.692605 6561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } }
I20260928 21:12:28.692718 6561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:28.692960 6562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4061bb912b55432b8bf8cb174d023b32. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } }
I20260928 21:12:28.693060 6562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:28.693341 6569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:28.694334 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:28.679887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33416 (local address 127.0.98.62:35485)
0928 21:12:28.680338 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:28.680354 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:28.680392 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:12:28.681140 (+ 748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:28.681162 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:28.681510 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:28.681891 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:28.682713 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:28.683738 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:28.684590 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:28.685122 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:28.685294 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:28.688017 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:28.688027 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:28.688047 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:28.688084 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:28.690076 (+ 1992us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:28.690651 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:28.690657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:28.690663 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:28.690840 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:28.691145 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:28.691148 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:28.691150 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:28.691658 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:28.691877 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:28.692192 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 21:12:28.692535 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":145,"threads_started":1}
I20260928 21:12:28.696683 6569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:28.702358 6569 catalog_manager.cc:1347] Generated new cluster ID: 143d04b7ee5c4aa78770c9adffcc3f08
I20260928 21:12:28.702430 6569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:28.716120 6569 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:28.717131 6569 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:28.724259 6569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Generated new TSK 0
I20260928 21:12:28.724974 6569 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 21:12:28.754184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:28.865159 6581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:28.865478 6581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:28.865552 6581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:28.869637 6581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:28.869716 6581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:28.869800 6581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:28.875353 6581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:28.876469 6581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:28.877590 6581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:28.887247 6586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:28.887270 6587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:28.887578 6581 server_base.cc:1034] running on GCE node
W20260928 21:12:28.887271 6589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:28.887986 6581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:28.888641 6581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:28.889851 6581 hybrid_clock.cc:648] HybridClock initialized: now 1790629948889787 us; error 42 us; skew 500 ppm
Sep 28 21:12:28 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:28.892938 6581 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 21:12:28.893891 6581 webserver.cc:460] Webserver started at http://127.0.98.1:37539/ using document root <none> and password file <none>
I20260928 21:12:28.894451 6581 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:28.894536 6581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:28.894737 6581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:28.896713 6581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "f53c51d9d9104357bfa28537aa822e7f"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "a10f92d8243e99b5356b79d0f487c351"
server_key_iv: "d8ca79860b8e18c0d942da99d983a0f7"
server_key_version: "encryptionkey@0"
I20260928 21:12:28.897324 6581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f53c51d9d9104357bfa28537aa822e7f"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "a10f92d8243e99b5356b79d0f487c351"
server_key_iv: "d8ca79860b8e18c0d942da99d983a0f7"
server_key_version: "encryptionkey@0"
I20260928 21:12:28.901240 6581 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:12:28.903538 6596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:28.904721 6581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:28.904862 6581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "f53c51d9d9104357bfa28537aa822e7f"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "a10f92d8243e99b5356b79d0f487c351"
server_key_iv: "d8ca79860b8e18c0d942da99d983a0f7"
server_key_version: "encryptionkey@0"
I20260928 21:12:28.904970 6581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:28.923012 6581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:28.926329 6581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:28.926540 6581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:28.927157 6581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:28.927997 6581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:28.928064 6581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:28.928128 6581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:28.928165 6581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:28.937057 6581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44579
I20260928 21:12:28.937137 6709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44579 every 8 connection(s)
I20260928 21:12:28.938045 6581 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:28.940936 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6581
I20260928 21:12:28.941082 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:28.941401 392 external_mini_cluster.cc:1428] Setting key 8b25b8f20e14b39f1f4153fadeade97b
Sep 28 21:12:28 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:28.954559 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:28.940159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38891 (local address 127.0.98.62:35485)
0928 21:12:28.940293 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:28.940296 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:28.941459 (+ 1163us) server_negotiation.cc:407] Connection header received
0928 21:12:28.943007 (+ 1548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:28.943011 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:28.943066 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:28.943132 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:28.944457 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:28.945036 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:28.946938 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:28.947201 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:28.947294 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:28.950614 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:28.950620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:28.950623 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:28.950655 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:28.952196 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:28.952815 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:28.952818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:28.952819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:28.952950 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:28.953310 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:28.953313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:28.953314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:28.953526 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:28.953630 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:28.954294 (+ 664us) server_negotiation.cc:299] Negotiation successful
0928 21:12:28.954426 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:12:28.955821 6712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:28.940665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:38891)
0928 21:12:28.941292 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:28.941341 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:28.942772 (+ 1431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:28.943314 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:28.943321 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:28.943707 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:28.944279 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:28.944304 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:28.945169 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:28.945175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:28.946743 (+ 1568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:28.946758 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:28.947379 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:28.947386 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:28.947633 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:28.948367 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:28.948395 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:28.950392 (+ 1997us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:28.952353 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:28.952360 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:28.952409 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:28.952719 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:28.953035 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:28.953038 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:28.953039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:28.953189 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:28.953641 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:28.953646 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:28.954006 (+ 360us) client_negotiation.cc:769] Sending connection context
0928 21:12:28.954217 (+ 211us) client_negotiation.cc:240] Negotiation successful
0928 21:12:28.954592 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":197,"threads_started":1}
I20260928 21:12:28.957350 6710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260928 21:12:28.957638 6710 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:28.958158 6710 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260928 21:12:28.959787 6519 ts_manager.cc:194] Registered new tserver with Master: f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579)
I20260928 21:12:28.961126 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38891
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 21:12:28.971925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:29.084064 6715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:29.084324 6715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:29.084371 6715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:29.088541 6715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:29.088608 6715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:29.088671 6715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:29.094303 6715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:29.095347 6715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:29.096340 6715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:29.105901 6720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.105885 6721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.105907 6723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.106272 6715 server_base.cc:1034] running on GCE node
I20260928 21:12:29.106719 6715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:29.107335 6715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:29.108511 6715 hybrid_clock.cc:648] HybridClock initialized: now 1790629949108482 us; error 54 us; skew 500 ppm
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:29.111261 6715 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 21:12:29.112134 6715 webserver.cc:460] Webserver started at http://127.0.98.2:37879/ using document root <none> and password file <none>
I20260928 21:12:29.112699 6715 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:29.112783 6715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:29.113039 6715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:29.115094 6715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "feda8c526d274210beb5977a68ba296d"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "82c6b95076b57041213206512f0c88c0"
server_key_iv: "729bfad237b4cbb4bf8beb3a813d3387"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.115658 6715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "feda8c526d274210beb5977a68ba296d"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "82c6b95076b57041213206512f0c88c0"
server_key_iv: "729bfad237b4cbb4bf8beb3a813d3387"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.119724 6715 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.001s
I20260928 21:12:29.122279 6730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.123566 6715 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:29.123704 6715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "feda8c526d274210beb5977a68ba296d"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "82c6b95076b57041213206512f0c88c0"
server_key_iv: "729bfad237b4cbb4bf8beb3a813d3387"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.123818 6715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:29.152659 6715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:29.156180 6715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:29.156411 6715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:29.157088 6715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:29.157966 6715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:29.158046 6715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.158134 6715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:29.158177 6715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.167261 6715 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43489
I20260928 21:12:29.167277 6843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43489 every 8 connection(s)
I20260928 21:12:29.168210 6715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:29.168758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6715
I20260928 21:12:29.168946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:29.169284 392 external_mini_cluster.cc:1428] Setting key a8ec937a5c9f5a6b0b182c7b0526a2ea
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:29.183760 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.170263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42811 (local address 127.0.98.62:35485)
0928 21:12:29.170402 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.170407 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.171490 (+ 1083us) server_negotiation.cc:407] Connection header received
0928 21:12:29.172911 (+ 1421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.172915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.172971 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.173080 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.174349 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.174872 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.176014 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.176240 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.176353 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.179513 (+ 3160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.179517 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.179519 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.179545 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.181113 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.181688 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.181691 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.181693 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.181795 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.182255 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.182260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.182263 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.182605 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.182731 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.183389 (+ 658us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.183551 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:12:29.184953 6846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.170813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:42811)
0928 21:12:29.171333 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.171395 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.172643 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.173264 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.173271 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.173645 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.174207 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.174221 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.174996 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.175000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.175889 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.175898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.176403 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.176407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.176573 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.177286 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.177311 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.179315 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.181222 (+ 1907us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.181227 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.181242 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.181527 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.181917 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.181923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.181928 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.182137 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.182776 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.182782 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.183107 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.183290 (+ 183us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.183688 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":148,"threads_started":1}
I20260928 21:12:29.186411 6844 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260928 21:12:29.186702 6844 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:29.187253 6844 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260928 21:12:29.188433 6519 ts_manager.cc:194] Registered new tserver with Master: feda8c526d274210beb5977a68ba296d (127.0.98.2:43489)
I20260928 21:12:29.189328 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42811
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 21:12:29.199262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:29.312273 6849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:29.312685 6849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:29.312760 6849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:29.317375 6849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:29.317441 6849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:29.317533 6849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:29.322988 6849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35485
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:29.324126 6849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:29.325263 6849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:29.335115 6857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.335290 6849 server_base.cc:1034] running on GCE node
W20260928 21:12:29.335101 6854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.335101 6855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.335793 6849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:29.336423 6849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:29.337581 6849 hybrid_clock.cc:648] HybridClock initialized: now 1790629949337562 us; error 31 us; skew 500 ppm
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:29.340281 6849 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 21:12:29.341292 6849 webserver.cc:460] Webserver started at http://127.0.98.3:44633/ using document root <none> and password file <none>
I20260928 21:12:29.341850 6849 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:29.341934 6849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:29.342156 6849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:29.344170 6849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "193c0b53f93243fbb0e499c350b63654"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "97fa1b3a640680d02aebf57c67adf9f8"
server_key_iv: "2a6cfb1403d159efafb099e38f1b3b89"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.344722 6849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "193c0b53f93243fbb0e499c350b63654"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "97fa1b3a640680d02aebf57c67adf9f8"
server_key_iv: "2a6cfb1403d159efafb099e38f1b3b89"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.348366 6849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:29.350783 6864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.351994 6849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:29.352120 6849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "193c0b53f93243fbb0e499c350b63654"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "97fa1b3a640680d02aebf57c67adf9f8"
server_key_iv: "2a6cfb1403d159efafb099e38f1b3b89"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.352226 6849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:29.372243 6849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:29.375466 6849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:29.375674 6849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:29.376268 6849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:29.377149 6849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:29.377214 6849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.377287 6849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:29.377326 6849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.386006 6977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38533 every 8 connection(s)
I20260928 21:12:29.385936 6849 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38533
I20260928 21:12:29.387071 6849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:29.396149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6849
I20260928 21:12:29.396292 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:29.396634 392 external_mini_cluster.cc:1428] Setting key bdd031104e2caafa00c1df564d87d3d2
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:29.401501 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.389074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44449 (local address 127.0.98.62:35485)
0928 21:12:29.389205 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.389209 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.390312 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 21:12:29.391728 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.391732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.391786 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.391920 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.393238 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.393827 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.394956 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.395177 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.395288 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.397902 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.397906 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.397908 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.397938 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.399423 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.399943 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.399947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.399948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.400023 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.400382 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.400385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.400387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.400569 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.400672 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.401210 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.401328 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:12:29.402289 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.389637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:44449)
0928 21:12:29.390168 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.390210 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.391521 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.392105 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.392111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.392547 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.393080 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.393094 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.393956 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.393964 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.394829 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.394839 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.395323 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.395328 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.395497 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.396063 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.396085 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.397745 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.399559 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.399564 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.399583 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.399833 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.400139 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.400142 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.400144 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.400283 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.400670 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.400678 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.400977 (+ 299us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.401157 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.401486 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":170,"threads_started":1}
I20260928 21:12:29.403331 6978 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260928 21:12:29.403589 6978 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:29.404064 6978 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260928 21:12:29.405032 6519 ts_manager.cc:194] Registered new tserver with Master: 193c0b53f93243fbb0e499c350b63654 (127.0.98.3:38533)
I20260928 21:12:29.405592 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44449
I20260928 21:12:29.410370 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260928 21:12:29.436129 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34725
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35485,127.0.98.61:34725
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:29.551240 6983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:29.551525 6983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:29.551566 6983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:29.555783 6983 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:29.555855 6983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:29.555873 6983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:29.555923 6983 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:29.555948 6983 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:29.561436 6983 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35485,127.0.98.61:34725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
W20260928 21:12:29.561674 6983 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:35485,127.0.98.61:34725'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260928 21:12:29.562600 6983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:29.563655 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:29.572356 6989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.572337 6991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.572337 6988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.572570 6983 server_base.cc:1034] running on GCE node
I20260928 21:12:29.573374 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:29.574563 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:29.575762 6983 hybrid_clock.cc:648] HybridClock initialized: now 1790629949575743 us; error 38 us; skew 500 ppm
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:29.578836 6983 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 21:12:29.579856 6983 webserver.cc:460] Webserver started at http://127.0.98.61:38521/ using document root <none> and password file <none>
I20260928 21:12:29.580495 6983 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:29.580592 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:29.580886 6983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:29.583196 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/instance:
uuid: "4b16a1c234244e14b558d76fc5958e29"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "de19fdb97403fc5f6cbfdc6956802602"
server_key_iv: "ec657ccad3487cf4dc1aeb8be9270dbe"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.583811 6983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal/instance:
uuid: "4b16a1c234244e14b558d76fc5958e29"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "de19fdb97403fc5f6cbfdc6956802602"
server_key_iv: "ec657ccad3487cf4dc1aeb8be9270dbe"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.587916 6983 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:29.590452 6998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.591728 6983 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:29.591871 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
uuid: "4b16a1c234244e14b558d76fc5958e29"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "de19fdb97403fc5f6cbfdc6956802602"
server_key_iv: "ec657ccad3487cf4dc1aeb8be9270dbe"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.592060 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:29.611568 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:29.614502 6983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:29.614660 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:29.621368 6983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34725
I20260928 21:12:29.621662 7050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34725 every 8 connection(s)
I20260928 21:12:29.622435 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
I20260928 21:12:29.625418 7051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:29.629503 7051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:12:29.632717 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 6983
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 21:12:29.644510 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.631412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.0.98.62:35485)
0928 21:12:29.631576 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.631580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.632594 (+ 1014us) server_negotiation.cc:407] Connection header received
0928 21:12:29.634028 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.634036 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.634106 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.634158 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.635481 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.636064 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.637287 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.637532 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.637659 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.640607 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.640610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.640612 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.640640 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.642392 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.643073 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.643078 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.643080 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.643169 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.643503 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.643507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.643508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.643732 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.643846 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.644223 (+ 377us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.644357 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:12:29.645759 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.633742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39662 (local address 127.0.98.61:34725)
0928 21:12:29.634125 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.634138 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.634214 (+ 76us) server_negotiation.cc:407] Connection header received
0928 21:12:29.634357 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.634361 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.634533 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.634724 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.635543 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.636612 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.637551 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.638013 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.638113 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.640907 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.640913 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.640924 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.640944 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.642601 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.643053 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.643065 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.643066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.643172 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.643444 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.643449 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.643450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.643829 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.644062 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.644233 (+ 171us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.644630 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":269,"spinlock_wait_cycles":4736,"thread_start_us":139,"threads_started":1}
I20260928 21:12:29.645759 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.631798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33428)
0928 21:12:29.632350 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.632450 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.633731 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.634431 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.634438 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.634813 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.635300 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.635329 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.636214 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.636223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.637152 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.637162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.637700 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.637704 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.637906 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.638575 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.638602 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.640423 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.642533 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.642542 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.642553 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.642803 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.643275 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.643278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.643281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.643406 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.643846 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.643854 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.643993 (+ 139us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.644180 (+ 187us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.644682 (+ 502us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":231,"thread_start_us":127,"threads_started":1}
I20260928 21:12:29.647291 7051 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 21:12:29.655262 7053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.647802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39670)
0928 21:12:29.648078 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.648111 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.648203 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.648426 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.648429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.648567 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.648660 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.648665 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.649432 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.649435 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.649968 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.649975 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.650371 (+ 396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.650373 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.650444 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.650888 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.650898 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.652190 (+ 1292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.653782 (+ 1592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.653784 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.653786 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.653994 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.654380 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.654382 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.654384 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.654473 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.654799 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.654801 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.654919 (+ 118us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.655010 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.655123 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156}
I20260928 21:12:29.655349 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.647919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39670 (local address 127.0.98.61:34725)
0928 21:12:29.648088 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.648094 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.648180 (+ 86us) server_negotiation.cc:407] Connection header received
0928 21:12:29.648305 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.648307 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.648337 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.648403 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.648754 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.649347 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.650062 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.650285 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.650361 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.652307 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.652310 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.652312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.652331 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.653685 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.654075 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.654078 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.654079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.654149 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.654559 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.654561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.654562 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.654720 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.654791 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.655071 (+ 280us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.655196 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:12:29.658779 7051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Bootstrap starting.
I20260928 21:12:29.661263 7051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:29.662010 7051 log.cc:824] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:29.664018 7051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: No bootstrap required, opened a new log
I20260928 21:12:29.667188 7051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } }
I20260928 21:12:29.667429 7051 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:29.667515 7051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b16a1c234244e14b558d76fc5958e29, State: Initialized, Role: FOLLOWER
I20260928 21:12:29.667950 7051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } }
I20260928 21:12:29.668437 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } } }
I20260928 21:12:29.668568 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:29.669284 7051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:12:29.675026 7068 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:12:29.675123 7068 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:12:29.675863 6983 master_runner.cc:428] Detected that this master 4b16a1c234244e14b558d76fc5958e29 is joining an existing cluster
I20260928 21:12:29.675915 6983 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:34725
I20260928 21:12:29.687464 7075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.678608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39676)
0928 21:12:29.678874 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.678886 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.678950 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.679652 (+ 702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.679655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.679867 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.679977 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.679984 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.680995 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.680998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.681878 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.681888 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.682595 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.682598 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.682726 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.683199 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.683214 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.683685 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.685842 (+ 2157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.685848 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.685850 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.686142 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.686401 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.686403 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.686405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.686499 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.686924 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.686927 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.687003 (+ 76us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.687111 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.687258 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":124,"threads_started":1}
I20260928 21:12:29.687526 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.678213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33432 (local address 127.0.98.62:35485)
0928 21:12:29.678939 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.678944 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.678958 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:29.678999 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.679002 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.679053 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.679132 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.679680 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.680507 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.681754 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.682096 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.682164 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.683668 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.683675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.683681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.683711 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.685558 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.686077 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.686081 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.686083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.686179 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.686530 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.686536 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.686541 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.686790 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.686889 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.687272 (+ 383us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.687380 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":642}
I20260928 21:12:29.687599 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.679300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39676 (local address 127.0.98.61:34725)
0928 21:12:29.679403 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.679407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.679419 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:29.679452 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.679455 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.679491 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.679564 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.680095 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.680876 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.681998 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.682322 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.682433 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.683813 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.683817 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.683819 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.683840 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.685732 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.686227 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.686232 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.686234 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.686314 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.686592 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.686595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.686600 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.686789 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.686863 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.687154 (+ 291us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.687400 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":212,"server-negotiator.queue_time_us":45,"spinlock_wait_cycles":8832}
I20260928 21:12:29.688269 7074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.678153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33432)
0928 21:12:29.678435 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.678450 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.678549 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.679182 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.679184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.679409 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.679546 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.679553 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.680644 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.680647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.681628 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.681637 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.682318 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.682321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.682483 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.683018 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.683031 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.683518 (+ 487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.685668 (+ 2150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.685673 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.685675 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.685969 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.686329 (+ 360us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.686334 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.686337 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.686431 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.686894 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.686897 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.686988 (+ 91us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.687100 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.687483 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":125,"thread_start_us":93,"threads_started":1}
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 21:12:29.704167 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.692899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39690)
0928 21:12:29.693265 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.693314 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.693438 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.693781 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.693786 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.694166 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.694276 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.694283 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.695226 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.695229 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.695988 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.695998 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.696605 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.696608 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.696695 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.697303 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.697315 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.698949 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.702137 (+ 3188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.702145 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.702151 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.702508 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.702865 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.702869 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.702871 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.702960 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.703471 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.703478 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.703631 (+ 153us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.703808 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.704013 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":86,"threads_started":1}
I20260928 21:12:29.704164 7055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.692899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39690 (local address 127.0.98.61:34725)
0928 21:12:29.693003 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.693007 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.693384 (+ 377us) server_negotiation.cc:407] Connection header received
0928 21:12:29.693580 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.693583 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.693634 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.693709 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.694382 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.695130 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.696154 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.696496 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.696587 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.699180 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.699186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.699191 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.699219 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.701871 (+ 2652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.702647 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.702651 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.702653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.702732 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.703055 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.703060 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.703063 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.703305 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.703395 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.703865 (+ 470us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.703988 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:12:29.705760 6519 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:34725: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4b16a1c234244e14b558d76fc5958e29"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 34725
}
attrs {
promote: true
}
}
dest_uuid: "4061bb912b55432b8bf8cb174d023b32"
cas_config_opid_index: -1
I20260928 21:12:29.706612 6519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } }
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260928 21:12:29.712520 7079 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4b16a1c234244e14b558d76fc5958e29 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } }
I20260928 21:12:29.713712 7025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:29.715122 7079 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:34725
I20260928 21:12:29.715333 7025 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Refusing update from remote peer 4061bb912b55432b8bf8cb174d023b32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260928 21:12:29.715356 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } }
I20260928 21:12:29.715445 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:29.715934 6983 master.cc:537] Master@127.0.98.61:34725 shutting down...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
I20260928 21:12:29.716548 7078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:12:29.721882 6503 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260928 21:12:29.722029 6983 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:29.722414 6983 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:29.722618 6983 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: stopping tablet replica
W20260928 21:12:29.722747 6503 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 -> Peer 4b16a1c234244e14b558d76fc5958e29 (127.0.98.61:34725): Couldn't send request to peer 4b16a1c234244e14b558d76fc5958e29. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:12:29.728127 6983 master.cc:552] Master@127.0.98.61:34725 shutdown complete.
I20260928 21:12:29.728189 6983 master_runner.cc:305] Clearing existing system tablet
I20260928 21:12:29.728499 6983 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:29.728564 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:29.732991 6983 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260928 21:12:29.733996 7084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.734216 6983 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:29.734525 6983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:29.734618 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
uuid: "4b16a1c234244e14b558d76fc5958e29"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "de19fdb97403fc5f6cbfdc6956802602"
server_key_iv: "ec657ccad3487cf4dc1aeb8be9270dbe"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.737303 6983 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:29.739622 6983 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:12:29.739725 6983 log.cc:1196] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260928 21:12:29.739950 6983 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Deleting consensus metadata
I20260928 21:12:29.740132 6983 master_runner.cc:315] Copying system tablet from 127.0.98.62:35485
I20260928 21:12:29.741266 6983 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35485
I20260928 21:12:29.750100 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.741922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33440 (local address 127.0.98.62:35485)
0928 21:12:29.742055 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.742059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.742268 (+ 209us) server_negotiation.cc:407] Connection header received
0928 21:12:29.742408 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.742411 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.742463 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.742554 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.743017 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.743542 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.744375 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.744610 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.744717 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.746236 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.746240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.746243 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.746277 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.748154 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.748689 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.748692 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.748694 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.748846 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:29.749162 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:29.749166 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:29.749168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:29.749428 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:29.749520 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:29.749795 (+ 275us) server_negotiation.cc:299] Negotiation successful
0928 21:12:29.749917 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:12:29.750372 7090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.741806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33440)
0928 21:12:29.742172 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.742184 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.742314 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.742561 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.742564 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.742811 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.742900 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.742906 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.743658 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.743663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.744250 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.744257 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.744721 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.744725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.744807 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.745454 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.745489 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.746018 (+ 529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.748275 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.748280 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.748282 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.748596 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.748938 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.748943 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:29.748945 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:29.749062 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:29.749520 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:29.749524 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:29.749626 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 21:12:29.749735 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:12:29.749943 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":82,"threads_started":1}
I20260928 21:12:29.751168 6539 tablet_copy_service.cc:140] P 4061bb912b55432b8bf8cb174d023b32: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4b16a1c234244e14b558d76fc5958e29 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33440)
I20260928 21:12:29.751387 6539 tablet_copy_service.cc:161] P 4061bb912b55432b8bf8cb174d023b32: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4b16a1c234244e14b558d76fc5958e29 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33440: session id = 4b16a1c234244e14b558d76fc5958e29-00000000000000000000000000000000
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260928 21:12:29.753111 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43063
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:12:29.753307 6539 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:12:29.755070 6983 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:12:29.757452 6983 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: tablet copy: Starting download of 0 data blocks...
I20260928 21:12:29.757653 6983 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: tablet copy: Starting download of 1 WAL segments...
I20260928 21:12:29.759546 6983 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:12:29.761775 6539 tablet_copy_service.cc:342] P 4061bb912b55432b8bf8cb174d023b32: Request end of tablet copy session 4b16a1c234244e14b558d76fc5958e29-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33440
I20260928 21:12:29.761910 6539 tablet_copy_service.cc:433] P 4061bb912b55432b8bf8cb174d023b32: ending tablet copy session 4b16a1c234244e14b558d76fc5958e29-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4b16a1c234244e14b558d76fc5958e29
I20260928 21:12:29.769017 6983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:29.770779 7092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.771070 7095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.771104 6983 server_base.cc:1034] running on GCE node
W20260928 21:12:29.772976 7093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.773250 6983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:29.773597 6983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:29.774739 6983 hybrid_clock.cc:648] HybridClock initialized: now 1790629949774749 us; error 44 us; skew 500 ppm
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:29.776362 6983 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260928 21:12:29.776880 6983 webserver.cc:460] Webserver started at http://127.0.98.61:39375/ using document root <none> and password file <none>
I20260928 21:12:29.777148 6983 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:29.777204 6983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:29.778319 6983 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:29.779264 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.779491 6983 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:29.779584 6983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
uuid: "4b16a1c234244e14b558d76fc5958e29"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "de19fdb97403fc5f6cbfdc6956802602"
server_key_iv: "ec657ccad3487cf4dc1aeb8be9270dbe"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.779685 6983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:29.791765 6983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:29.794202 6983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:29.798988 6983 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34725
I20260928 21:12:29.799566 6983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
I20260928 21:12:29.799932 7154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34725 every 8 connection(s)
I20260928 21:12:29.801194 7155 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:12:29.802018 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Bootstrap starting.
I20260928 21:12:29.810156 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:12:29.810441 7155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Bootstrap complete.
I20260928 21:12:29.810916 7155 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } }
I20260928 21:12:29.811048 7155 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4b16a1c234244e14b558d76fc5958e29, State: Initialized, Role: LEARNER
I20260928 21:12:29.811162 7155 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } }
I20260928 21:12:29.811481 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } }
I20260928 21:12:29.811641 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:12:29.811661 7155 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:29.814543 7169 catalog_manager.cc:1259] Loaded cluster ID: 143d04b7ee5c4aa78770c9adffcc3f08
I20260928 21:12:29.814718 7169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: loading cluster ID for follower catalog manager: success
I20260928 21:12:29.816231 7169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: acquiring CA information for follower catalog manager: success
I20260928 21:12:29.817365 7169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:12:29.905607 7091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:29.905936 7091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:29.906023 7091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:29.910250 7091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:29.910339 7091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:29.910377 7091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:29.910429 7091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:29.910478 7091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:29.916635 7091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:43063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:29.917812 7091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:29.918856 7091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:29.927762 7175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.927712 7174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:29.927687 7177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:29.928098 7091 server_base.cc:1034] running on GCE node
I20260928 21:12:29.928690 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:29.929916 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:29.931082 7091 hybrid_clock.cc:648] HybridClock initialized: now 1790629949931066 us; error 29 us; skew 500 ppm
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:29.933988 7091 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 21:12:29.934957 7091 webserver.cc:460] Webserver started at http://127.0.98.60:41839/ using document root <none> and password file <none>
I20260928 21:12:29.935530 7091 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:29.935595 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:29.935807 7091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:29.938012 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/instance:
uuid: "4da322c2a8994151876290659f8ecb04"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "a681492577e9e696a36963bfaca453cd"
server_key_iv: "7485e56827ff61dc2ad8a20a663c1ee6"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.938591 7091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal/instance:
uuid: "4da322c2a8994151876290659f8ecb04"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "a681492577e9e696a36963bfaca453cd"
server_key_iv: "7485e56827ff61dc2ad8a20a663c1ee6"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.942420 7091 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:29.945046 7184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:29.946449 7091 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:12:29.946622 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
uuid: "4da322c2a8994151876290659f8ecb04"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "a681492577e9e696a36963bfaca453cd"
server_key_iv: "7485e56827ff61dc2ad8a20a663c1ee6"
server_key_version: "encryptionkey@0"
I20260928 21:12:29.946750 7091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:29.963650 6710 heartbeater.cc:503] Master 127.0.98.62:35485 was elected leader, sending a full tablet report...
I20260928 21:12:29.966603 7091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:29.970350 7091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:29.970584 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:29.978785 7091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43063
I20260928 21:12:29.979158 7236 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43063 every 8 connection(s)
I20260928 21:12:29.979987 7091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
I20260928 21:12:29.982954 7237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:29.987712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 7091
I20260928 21:12:29.987505 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 21:12:29 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 21:12:30.002290 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.989752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33452 (local address 127.0.98.62:35485)
0928 21:12:29.989883 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.989888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.990270 (+ 382us) server_negotiation.cc:407] Connection header received
0928 21:12:29.990993 (+ 723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.990996 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.991046 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.991130 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.992356 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.993296 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.994947 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.995261 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.995382 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.998171 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.998175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.998178 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.998210 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:29.999869 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.000412 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.000415 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.000416 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.000489 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.000917 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.000921 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.000922 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.001195 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.001272 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.001958 (+ 686us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.002109 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:12:30.003443 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.989412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47262 (local address 127.0.98.60:43063)
0928 21:12:29.989997 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:29.990021 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:29.990054 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:12:29.990746 (+ 692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:29.990764 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:29.990994 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:29.991179 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:29.992199 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.993546 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.994801 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.995348 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.995563 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.998660 (+ 3097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:29.998670 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:29.998688 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:29.998716 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.000657 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.001184 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.001189 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.001191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.001299 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.001577 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.001580 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.001582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.001852 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.002026 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.002261 (+ 235us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.002379 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":426,"thread_start_us":144,"threads_started":1}
I20260928 21:12:30.003443 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:29.989585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33452)
0928 21:12:29.990116 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:29.990167 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:29.990770 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:29.991213 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:29.991219 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:29.991745 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:29.992195 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.992227 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.993430 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.993432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.994782 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:29.994793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:29.995471 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:29.995476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:29.995640 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:29.996254 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:29.996278 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:29.998010 (+ 1732us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:29.999993 (+ 1983us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:29.999999 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.000011 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.000313 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.000587 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.000591 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.000593 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.000775 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.001299 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.001307 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.001666 (+ 359us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.001889 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.002271 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":5,"thread_start_us":107,"threads_started":1}
I20260928 21:12:30.004734 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260928 21:12:30.013584 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.005383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.0.98.61:34725)
0928 21:12:30.005662 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.005665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.005676 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:12:30.005711 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.005713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.005778 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.005878 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.006285 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.006947 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.007691 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.007899 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.007974 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.010330 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.010333 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.010335 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.010356 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.011844 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.012276 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.012279 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.012280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.012353 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.012695 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.012700 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.012702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.012936 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.013039 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.013301 (+ 262us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.013421 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":106,"threads_started":1}
I20260928 21:12:30.013581 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.005275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39706)
0928 21:12:30.005383 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.005397 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.005475 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.005931 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.005934 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.006092 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.006183 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.006188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.007042 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.007044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.007594 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.007601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.008043 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.008051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.008186 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.008733 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.008744 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.010189 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.011986 (+ 1797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.011989 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.011990 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.012189 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.012496 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.012500 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.012502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.012597 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.013028 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.013031 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.013130 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.013241 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.013381 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:12:30.014628 7237 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629949, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 21:12:30.022909 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.015073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47266)
0928 21:12:30.015196 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.015208 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.015270 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.015487 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.015489 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.015620 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.015689 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.015693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.016412 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.016414 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.017035 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.017043 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.017486 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.017489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.017545 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.017998 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.018009 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.019169 (+ 1160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.021492 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.021494 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.021504 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.021694 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.021950 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.021952 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.021953 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.022030 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.022436 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.022438 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.022517 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.022618 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.022722 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:12:30.022912 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.015185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47266 (local address 127.0.98.60:43063)
0928 21:12:30.015310 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.015313 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.015324 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:12:30.015351 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.015366 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.015398 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.015462 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.015832 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.016321 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.017143 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.017390 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.017477 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.019306 (+ 1829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.019310 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.019312 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.019331 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.021358 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.021788 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.021790 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.021791 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.021865 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.022141 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.022144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.022145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.022323 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.022411 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.022674 (+ 263us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.022769 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:12:30.026134 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Bootstrap starting.
I20260928 21:12:30.029182 7237 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:30.029978 7237 log.cc:824] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:30.032045 7237 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: No bootstrap required, opened a new log
I20260928 21:12:30.035445 7237 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } }
I20260928 21:12:30.035717 7237 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:30.035804 7237 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da322c2a8994151876290659f8ecb04, State: Initialized, Role: FOLLOWER
I20260928 21:12:30.036281 7237 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } }
I20260928 21:12:30.036813 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } } }
I20260928 21:12:30.037010 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:30.037653 7237 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: configured and running, proceeding with master startup.
W20260928 21:12:30.042939 7254 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:12:30.043042 7254 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:12:30.046785 7091 master_runner.cc:428] Detected that this master 4da322c2a8994151876290659f8ecb04 is joining an existing cluster
I20260928 21:12:30.046875 7091 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:43063
I20260928 21:12:30.057400 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6581
I20260928 21:12:30.064774 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:44579
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:35485
--webserver_port=37539
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:12:30.079301 7260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.050177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33466)
0928 21:12:30.050508 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.050522 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.050702 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.051133 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.051135 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.051772 (+ 637us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.051891 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.051897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.053110 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.053147 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.054246 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.054256 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.055308 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.055315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.055421 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.073513 (+ 18092us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.073539 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.074302 (+ 763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.077600 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.077603 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.077605 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.077837 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.078157 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.078162 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.078164 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.078244 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.078704 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.078707 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.078815 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.078911 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.079072 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":43,"thread_start_us":106,"threads_started":1}
I20260928 21:12:30.080132 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.049991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33466 (local address 127.0.98.62:35485)
0928 21:12:30.050131 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.050136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.050630 (+ 494us) server_negotiation.cc:407] Connection header received
0928 21:12:30.050826 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.050833 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.050893 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.050951 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.052036 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.052915 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.054392 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.054727 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.054796 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.074547 (+ 19751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.074553 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.074556 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.074589 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.077462 (+ 2873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.077966 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.077970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.077971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.078054 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.078339 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.078342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.078344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.078567 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.078663 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.079839 (+ 1176us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.079976 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:12:30.080561 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.050576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47270 (local address 127.0.98.60:43063)
0928 21:12:30.050678 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.050682 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.051187 (+ 505us) server_negotiation.cc:407] Connection header received
0928 21:12:30.051466 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.051472 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.051520 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.051608 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.052107 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.052914 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.054537 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.054875 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.054947 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.074631 (+ 19684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.074638 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.074642 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.074672 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.076805 (+ 2133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.078510 (+ 1705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.078514 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.078516 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.078585 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.079177 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.079181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.079182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.079375 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.079441 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.080303 (+ 862us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.080424 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":41}
I20260928 21:12:30.081374 7262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.050760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47270)
0928 21:12:30.051082 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.051132 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.051355 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.051618 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.051621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.051890 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.052009 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.052017 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.053070 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.053147 (+ 77us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.054440 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.054447 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.055116 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.055120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.055259 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.055733 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.055743 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.074437 (+ 18694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.077757 (+ 3320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.077760 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.077762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.078000 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.078674 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.078676 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.078678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.078744 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.079497 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.079499 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.079582 (+ 83us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.080952 (+ 1370us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.081207 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":40,"thread_start_us":80,"threads_started":1}
I20260928 21:12:30.081863 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.050055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.0.98.61:34725)
0928 21:12:30.050366 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.050371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.050597 (+ 226us) server_negotiation.cc:407] Connection header received
0928 21:12:30.050787 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.050794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.050856 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.050913 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.051802 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.052639 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.053917 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.054281 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.054374 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.075235 (+ 20861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.075242 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.075248 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.075287 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.077566 (+ 2279us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.078837 (+ 1271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.078841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.078843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.078920 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.079801 (+ 881us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.079804 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.079806 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.080028 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.080093 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.081585 (+ 1492us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.081719 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260928 21:12:30.082408 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.049923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39712)
0928 21:12:30.050503 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.050519 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.050673 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.051064 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.051067 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.051491 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.051645 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.051653 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.052786 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.052789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.053773 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.053784 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.054538 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.054542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.054674 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.055284 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.055321 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.075010 (+ 19689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.078107 (+ 3097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.078110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.078112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.078312 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.079026 (+ 714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.079029 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.079031 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.079706 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.080160 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.080163 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.080802 (+ 639us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.082078 (+ 1276us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.082222 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":24,"spinlock_wait_cycles":3968,"thread_start_us":153,"threads_started":1}
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629948, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 21:12:30.092269 7239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.084560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47282 (local address 127.0.98.60:43063)
0928 21:12:30.084671 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.084676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.084689 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:30.084756 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.084765 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.084805 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.084930 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.085334 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.085854 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.086634 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.086871 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.086980 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.089096 (+ 2116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.089099 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.089101 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.089121 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.090541 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.091014 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.091020 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.091021 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.091100 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.091382 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.091388 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.091392 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.091571 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.091718 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.092000 (+ 282us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.092103 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:12:30.092269 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.084336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47282)
0928 21:12:30.084513 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.084527 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.084611 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.084946 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.084949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.085130 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.085213 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.085219 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.085964 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.085969 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.086515 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.086523 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.086982 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.086985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.087086 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.087582 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.087594 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.088955 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.090699 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.090704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.090707 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.090917 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.091192 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.091197 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.091199 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.091287 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.091706 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.091709 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.091815 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.091947 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.092106 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260928 21:12:30.093673 6519 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:43063: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4da322c2a8994151876290659f8ecb04"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 43063
}
attrs {
promote: true
}
}
dest_uuid: "4061bb912b55432b8bf8cb174d023b32"
cas_config_opid_index: 5
I20260928 21:12:30.094256 6519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } }
I20260928 21:12:30.095309 7078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4da322c2a8994151876290659f8ecb04 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.096652 7211 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:30.096999 7078 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:43063
I20260928 21:12:30.097175 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.097270 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.098109 7211 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Refusing update from remote peer 4061bb912b55432b8bf8cb174d023b32: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:12:30.098677 7079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:30.098815 7091 master.cc:537] Master@127.0.98.60:43063 shutting down...
W20260928 21:12:30.100942 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 -> Peer 4da322c2a8994151876290659f8ecb04 (127.0.98.60:43063): Couldn't send request to peer 4da322c2a8994151876290659f8ecb04. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260928 21:12:30.104475 7091 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:30.104961 7091 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:30.105165 7091 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: stopping tablet replica
I20260928 21:12:30.108453 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.095480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39714)
0928 21:12:30.103212 (+ 7732us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.103233 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.103708 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.104189 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.104195 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.104219 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.104431 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.104442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.105513 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.105517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.106870 (+ 1353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.106883 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.107898 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.107902 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.107994 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.108049 (+ 55us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.108133 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.108214 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7625}
I20260928 21:12:30.109030 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.095739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39714 (local address 127.0.98.61:34725)
0928 21:12:30.100639 (+ 4900us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.100644 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.103319 (+ 2675us) server_negotiation.cc:407] Connection header received
0928 21:12:30.103855 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.103861 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.103937 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.104032 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.104577 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.105366 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.107028 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.107546 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.107908 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.108061 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.108687 (+ 626us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.108778 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4779}
I20260928 21:12:30.109680 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 LEARNER]: Refusing update from remote peer 4061bb912b55432b8bf8cb174d023b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260928 21:12:30.110302 7079 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260928 21:12:30.112401 7129 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 4da322c2a8994151876290659f8ecb04 (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.114444 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4061bb912b55432b8bf8cb174d023b32. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.114562 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:12:30.114657 7158 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: true } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.114737 7158 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:12:30.115054 7091 master.cc:552] Master@127.0.98.60:43063 shutdown complete.
I20260928 21:12:30.115152 7091 master_runner.cc:305] Clearing existing system tablet
I20260928 21:12:30.115419 7079 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: attempting to promote NON_VOTER 4b16a1c234244e14b558d76fc5958e29 to VOTER
I20260928 21:12:30.115525 7091 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:30.115641 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:30.115957 7079 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } }
I20260928 21:12:30.117522 7129 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 LEARNER]: Refusing update from remote peer 4061bb912b55432b8bf8cb174d023b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260928 21:12:30.118016 7078 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260928 21:12:30.118331 7091 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:30.119266 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.119510 7091 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:30.119637 7091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:30.119673 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
uuid: "4da322c2a8994151876290659f8ecb04"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "a681492577e9e696a36963bfaca453cd"
server_key_iv: "7485e56827ff61dc2ad8a20a663c1ee6"
server_key_version: "encryptionkey@0"
I20260928 21:12:30.119807 7078 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b16a1c234244e14b558d76fc5958e29 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.121730 7078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.121830 7078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.121775 7091 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:30.121958 7079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.122032 7079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.117918 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.117141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47284)
0928 21:12:30.117750 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.117832 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531}
I20260928 21:12:30.123246 7076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.122687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47286)
0928 21:12:30.123051 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.123130 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43063: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":98,"threads_started":1}
W20260928 21:12:30.123332 6504 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 -> Peer 4da322c2a8994151876290659f8ecb04 (127.0.98.60:43063): Couldn't send request to peer 4da322c2a8994151876290659f8ecb04. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:43063: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:12:30.124258 7091 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260928 21:12:30.124347 7091 log.cc:1196] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260928 21:12:30.124539 7091 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Deleting consensus metadata
I20260928 21:12:30.125211 7128 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4b16a1c234244e14b558d76fc5958e29 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.126704 7091 master_runner.cc:315] Copying system tablet from 127.0.98.62:35485
I20260928 21:12:30.128039 7091 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:35485
I20260928 21:12:30.128464 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.128593 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:30.136518 7275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.128589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33470)
0928 21:12:30.129225 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.129239 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.129384 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.129669 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.129671 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.129886 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.129979 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.129985 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.130757 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.130762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.131375 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.131384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.131903 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.131907 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.132018 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.132615 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.132630 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.133107 (+ 477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.134944 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.134948 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.134951 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.135159 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.135449 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.135453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.135456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.135542 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.135941 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.135944 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.136052 (+ 108us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.136163 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.136322 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":95,"threads_started":1}
I20260928 21:12:30.136507 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.129027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33470 (local address 127.0.98.62:35485)
0928 21:12:30.129171 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.129177 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.129328 (+ 151us) server_negotiation.cc:407] Connection header received
0928 21:12:30.129495 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.129499 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.129559 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.129649 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.130102 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.130639 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.131501 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.131787 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.131881 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.133254 (+ 1373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.133258 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.133261 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.133288 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.134819 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.135264 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.135267 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.135269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.135352 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.135640 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.135643 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.135645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.135832 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.135951 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.136217 (+ 266us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.136333 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:12:30.137442 6539 tablet_copy_service.cc:140] P 4061bb912b55432b8bf8cb174d023b32: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4da322c2a8994151876290659f8ecb04 ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33470)
I20260928 21:12:30.137562 6539 tablet_copy_service.cc:161] P 4061bb912b55432b8bf8cb174d023b32: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4da322c2a8994151876290659f8ecb04 at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33470: session id = 4da322c2a8994151876290659f8ecb04-00000000000000000000000000000000
I20260928 21:12:30.138612 6539 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Tablet Copy: opened 0 blocks and 1 log segments
I20260928 21:12:30.140040 7091 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260928 21:12:30.142226 7091 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: tablet copy: Starting download of 0 data blocks...
I20260928 21:12:30.142445 7091 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: tablet copy: Starting download of 1 WAL segments...
I20260928 21:12:30.143914 7091 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260928 21:12:30.145514 6539 tablet_copy_service.cc:342] P 4061bb912b55432b8bf8cb174d023b32: Request end of tablet copy session 4da322c2a8994151876290659f8ecb04-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:33470
I20260928 21:12:30.145619 6539 tablet_copy_service.cc:433] P 4061bb912b55432b8bf8cb174d023b32: ending tablet copy session 4da322c2a8994151876290659f8ecb04-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4da322c2a8994151876290659f8ecb04
I20260928 21:12:30.152645 7091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:30.154356 7279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.157006 7276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:30.157243 7091 server_base.cc:1034] running on GCE node
W20260928 21:12:30.161690 7277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:30.161972 7091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:30.162392 7091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:30.163560 7091 hybrid_clock.cc:648] HybridClock initialized: now 1790629950163550 us; error 29 us; skew 500 ppm
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:30.165179 7091 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260928 21:12:30.165686 7091 webserver.cc:460] Webserver started at http://127.0.98.60:35197/ using document root <none> and password file <none>
I20260928 21:12:30.165903 7091 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:30.165957 7091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:30.169030 7091 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:12:30.170104 7286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.170351 7091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:30.170408 7091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
uuid: "4da322c2a8994151876290659f8ecb04"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "a681492577e9e696a36963bfaca453cd"
server_key_iv: "7485e56827ff61dc2ad8a20a663c1ee6"
server_key_version: "encryptionkey@0"
I20260928 21:12:30.170477 7091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:30.179003 7091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:30.181272 7091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:30.186192 7091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:43063
I20260928 21:12:30.186751 7091 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
I20260928 21:12:30.188143 7339 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:12:30.189074 7339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Bootstrap starting.
I20260928 21:12:30.191464 6844 heartbeater.cc:503] Master 127.0.98.62:35485 was elected leader, sending a full tablet report...
I20260928 21:12:30.198282 7339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:12:30.198618 7339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: Bootstrap complete.
I20260928 21:12:30.198632 7338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:43063 every 8 connection(s)
I20260928 21:12:30.199167 7339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } }
I20260928 21:12:30.199306 7339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4da322c2a8994151876290659f8ecb04, State: Initialized, Role: LEARNER
I20260928 21:12:30.199460 7339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } }
I20260928 21:12:30.199808 7342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.199975 7339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:30.199977 7342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: This master's current role is: LEARNER
I20260928 21:12:30.203316 7353 catalog_manager.cc:1259] Loaded cluster ID: 143d04b7ee5c4aa78770c9adffcc3f08
I20260928 21:12:30.203433 7353 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: loading cluster ID for follower catalog manager: success
I20260928 21:12:30.205075 7353 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: acquiring CA information for follower catalog manager: success
I20260928 21:12:30.206182 7353 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260928 21:12:30.228020 7263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:30.228355 7263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:30.228417 7263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:30.234489 7263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:30.234568 7263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:30.234661 7263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:30.242719 7263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:44579
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37539
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:30.244655 7263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:30.245995 7263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:30.257905 7359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.257905 7358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.257918 7361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:30.258229 7263 server_base.cc:1034] running on GCE node
I20260928 21:12:30.258773 7263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:30.259460 7263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:30.260632 7263 hybrid_clock.cc:648] HybridClock initialized: now 1790629950260593 us; error 51 us; skew 500 ppm
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:30.263425 7263 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 21:12:30.264323 7263 webserver.cc:460] Webserver started at http://127.0.98.1:37539/ using document root <none> and password file <none>
I20260928 21:12:30.264943 7263 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:30.265045 7263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:30.268976 7263 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:30.271255 7368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.272578 7263 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:30.272718 7263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "f53c51d9d9104357bfa28537aa822e7f"
format_stamp: "Formatted at 2026-09-28 21:12:28 on dist-test-slave-v7xz"
server_key: "a10f92d8243e99b5356b79d0f487c351"
server_key_iv: "d8ca79860b8e18c0d942da99d983a0f7"
server_key_version: "encryptionkey@0"
I20260928 21:12:30.273398 7263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:30.294124 7263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:30.297237 7263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:30.297452 7263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:30.298055 7263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:30.298923 7263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:30.298995 7263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.299067 7263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:30.299109 7263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.308151 7263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44579
I20260928 21:12:30.308207 7481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44579 every 8 connection(s)
I20260928 21:12:30.309257 7263 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:30.315042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 7263
I20260928 21:12:30.315289 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6715
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 21:12:30.325469 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:43489
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:35485
--webserver_port=37879
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:12:30.328969 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.313499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47961 (local address 127.0.98.61:34725)
0928 21:12:30.313624 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.313629 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.313644 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:30.314824 (+ 1180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.314828 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.314888 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.314965 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.316757 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.317599 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.319634 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.319979 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.320051 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.324337 (+ 4286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.324340 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.324343 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.324372 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.327190 (+ 2818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327649 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.327653 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.327654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.327726 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.328046 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.328049 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.328051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.328255 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.328328 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.328668 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.328781 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:12:30.328991 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.311078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43295 (local address 127.0.98.62:35485)
0928 21:12:30.311216 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.311219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.313031 (+ 1812us) server_negotiation.cc:407] Connection header received
0928 21:12:30.314601 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.314605 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.314665 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.314760 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.316782 (+ 2022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.317615 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.319635 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.319969 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.320048 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.324283 (+ 4235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.324286 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.324291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.324321 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.326155 (+ 1834us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.326858 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.326864 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.326866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.326972 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327414 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.327420 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.327423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.327685 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.327763 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.328570 (+ 807us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.328715 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:12:30.330024 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.313378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.98.1:51043)
0928 21:12:30.314755 (+ 1377us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.317434 (+ 2679us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.317541 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.317954 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.317957 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.318184 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.318343 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.318354 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.319811 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.319818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.321431 (+ 1613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.321445 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.322317 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.322323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.322452 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.322992 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.323013 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.324604 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.326719 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.326724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.326727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.326963 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.327279 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327283 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.327285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.327392 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.327865 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.327868 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.328247 (+ 379us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.328469 (+ 222us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.328869 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1311,"mutex_wait_us":6,"spinlock_wait_cycles":8320,"thread_start_us":81,"threads_started":1}
I20260928 21:12:30.330027 7486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.311499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:43295)
0928 21:12:30.312816 (+ 1317us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.312925 (+ 109us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.314303 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.315083 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.315093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.315678 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.316580 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.316621 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.317766 (+ 1145us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.317770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.319497 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.319507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.320246 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.320250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.320551 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.321407 (+ 856us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.321435 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.324077 (+ 2642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.326293 (+ 2216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.326302 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.326322 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.326721 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.327104 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327109 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.327112 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.327300 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.327849 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.327858 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.328251 (+ 393us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.329142 (+ 891us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.329310 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":443,"spinlock_wait_cycles":12032,"thread_start_us":1001,"threads_started":1}
I20260928 21:12:30.330965 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.313739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51043 (local address 127.0.98.60:43063)
0928 21:12:30.313993 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.313997 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.317640 (+ 3643us) server_negotiation.cc:407] Connection header received
0928 21:12:30.317695 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.317697 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.317751 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.317830 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.318490 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.319670 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.321583 (+ 1913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.322038 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.322156 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.324728 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.324733 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.324736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.324767 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.326586 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327082 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.327087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.327092 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.327176 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327512 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.327516 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.327517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.327741 (+ 224us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.327830 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.330639 (+ 2809us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.330782 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":86,"threads_started":1}
I20260928 21:12:30.331350 7483 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260928 21:12:30.331640 7483 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.331677 7482 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43063
I20260928 21:12:30.331804 7482 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.331952 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.313128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.98.1:47961)
0928 21:12:30.313399 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.313415 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.314304 (+ 889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.315108 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.315111 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.315829 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.316579 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.316601 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.317766 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.317770 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.319497 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.319507 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.320246 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.320250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.320551 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.321457 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.321468 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.324077 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.327292 (+ 3215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327295 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.327297 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.327541 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.327839 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.327845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.327848 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.327932 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.328397 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.328400 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.328500 (+ 100us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.329576 (+ 1076us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.329713 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":323,"spinlock_wait_cycles":3584,"thread_start_us":94,"threads_started":1}
I20260928 21:12:30.332360 7482 heartbeater.cc:511] Master 127.0.98.60:43063 requested a full tablet report, sending...
I20260928 21:12:30.332357 7483 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260928 21:12:30.332818 7484 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34725
I20260928 21:12:30.332912 7484 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.333067 7484 heartbeater.cc:511] Master 127.0.98.61:34725 requested a full tablet report, sending...
I20260928 21:12:30.334075 6519 ts_manager.cc:194] Re-registered known tserver with Master: f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579)
I20260928 21:12:30.334232 7119 ts_manager.cc:194] Registered new tserver with Master: f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579)
I20260928 21:12:30.334532 7303 ts_manager.cc:194] Registered new tserver with Master: f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579)
I20260928 21:12:30.334759 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:43295
I20260928 21:12:30.407698 6978 heartbeater.cc:503] Master 127.0.98.62:35485 was elected leader, sending a full tablet report...
W20260928 21:12:30.444767 7490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:30.445109 7490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:30.445154 7490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:30.449237 7490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:30.449295 7490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:30.449354 7490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:30.454831 7490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:43489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37879
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:30.455950 7490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:30.457105 7490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:30.467399 7496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.467326 7498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.467370 7495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:30.467542 7490 server_base.cc:1034] running on GCE node
I20260928 21:12:30.468070 7490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:30.469081 7490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:30.470302 7490 hybrid_clock.cc:648] HybridClock initialized: now 1790629950470274 us; error 49 us; skew 500 ppm
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:30.473351 7490 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 21:12:30.474304 7490 webserver.cc:460] Webserver started at http://127.0.98.2:37879/ using document root <none> and password file <none>
I20260928 21:12:30.474879 7490 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:30.474987 7490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:30.479090 7490 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:30.481604 7505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.482894 7490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:30.483022 7490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "feda8c526d274210beb5977a68ba296d"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "82c6b95076b57041213206512f0c88c0"
server_key_iv: "729bfad237b4cbb4bf8beb3a813d3387"
server_key_version: "encryptionkey@0"
I20260928 21:12:30.483582 7490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:30.500876 7490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:30.504578 7490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:30.504783 7490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:30.505415 7490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:30.506259 7490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:30.506313 7490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.506359 7490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:30.506378 7490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.515944 7490 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43489
I20260928 21:12:30.515964 7618 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43489 every 8 connection(s)
I20260928 21:12:30.517081 7490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:30.523134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 7490
I20260928 21:12:30.523327 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6849
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:30.530891 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38533
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:35485
--webserver_port=44633
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:12:30.536217 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.518989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51321 (local address 127.0.98.61:34725)
0928 21:12:30.519118 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.519123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.520531 (+ 1408us) server_negotiation.cc:407] Connection header received
0928 21:12:30.521555 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.521558 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.521615 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.521694 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.523318 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.524207 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.526133 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.526469 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.526564 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.531553 (+ 4989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.531557 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.531559 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.531594 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.534255 (+ 2661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534727 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.534730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.534731 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.534819 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.535128 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.535134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.535139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.535352 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.535431 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.535922 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.536042 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:12:30.536334 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.520901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34789 (local address 127.0.98.62:35485)
0928 21:12:30.521025 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.521028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.521042 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:30.522002 (+ 960us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.522006 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.522063 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.522152 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.526472 (+ 4320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.527400 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.528538 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.528972 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.529038 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.531822 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.531827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.531830 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.531863 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.533720 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534334 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.534341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.534345 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.534437 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534873 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.534879 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.534884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.535142 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.535224 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.535951 (+ 727us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.536097 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:12:30.537343 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.519509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.98.2:51321)
0928 21:12:30.520440 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.520453 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.521312 (+ 859us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.521909 (+ 597us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.521916 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.522471 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.523150 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.523163 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.524312 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.524314 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.525993 (+ 1679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.526003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.526739 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.526743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.527015 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.527976 (+ 961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.527991 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.531333 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.534370 (+ 3037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.534375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.534618 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.534937 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534940 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.534942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.535024 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.535486 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.535488 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.535620 (+ 132us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.536381 (+ 761us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.536510 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":846,"mutex_wait_us":313,"spinlock_wait_cycles":49792,"thread_start_us":156,"threads_started":1}
I20260928 21:12:30.537352 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.520714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:34789)
0928 21:12:30.520979 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.520992 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.521348 (+ 356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.522179 (+ 831us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.522182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.522506 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.523139 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.523158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.527505 (+ 4347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.527510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.528389 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.528399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.529171 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.529174 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.529237 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.529740 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.529751 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.531660 (+ 1909us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.533850 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.533857 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.533862 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.534205 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.534550 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534556 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.534561 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.534760 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.535303 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.535309 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.535634 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.536631 (+ 997us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.536816 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":94,"spinlock_wait_cycles":61184,"thread_start_us":122,"threads_started":1}
I20260928 21:12:30.537617 7623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.519509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.98.2:59417)
0928 21:12:30.520059 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.520102 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.521312 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.521908 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.521915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.522472 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.523139 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.523158 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.524285 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.524291 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.525993 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.526003 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.526739 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.526742 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.527016 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.527858 (+ 842us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.527885 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.531329 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.533712 (+ 2383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.533720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.533739 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.534205 (+ 466us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.534546 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534552 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.534555 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.534753 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.535251 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.535258 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.535620 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.535841 (+ 221us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.536182 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":334,"spinlock_wait_cycles":36352,"thread_start_us":674,"threads_started":1}
I20260928 21:12:30.538741 7621 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34725
I20260928 21:12:30.538998 7621 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.539011 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.518995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59417 (local address 127.0.98.60:43063)
0928 21:12:30.519118 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.519123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.520226 (+ 1103us) server_negotiation.cc:407] Connection header received
0928 21:12:30.521710 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.521715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.521772 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.521868 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.523318 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.524114 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.526133 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.526490 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.526564 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.531537 (+ 4973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.531543 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.531549 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.531589 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.533567 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534331 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.534334 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.534337 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.534438 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.534856 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.534862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.534867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.535111 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.535198 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.538712 (+ 3514us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.538869 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:30.539575 7621 heartbeater.cc:511] Master 127.0.98.61:34725 requested a full tablet report, sending...
I20260928 21:12:30.539690 7619 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43063
I20260928 21:12:30.539794 7619 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.539999 7619 heartbeater.cc:511] Master 127.0.98.60:43063 requested a full tablet report, sending...
I20260928 21:12:30.538746 7620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260928 21:12:30.540261 7620 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.540593 7620 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260928 21:12:30.540802 7119 ts_manager.cc:194] Registered new tserver with Master: feda8c526d274210beb5977a68ba296d (127.0.98.2:43489)
I20260928 21:12:30.540861 7303 ts_manager.cc:194] Registered new tserver with Master: feda8c526d274210beb5977a68ba296d (127.0.98.2:43489)
I20260928 21:12:30.541303 6519 ts_manager.cc:194] Re-registered known tserver with Master: feda8c526d274210beb5977a68ba296d (127.0.98.2:43489)
I20260928 21:12:30.542196 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:34789
I20260928 21:12:30.547833 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.543687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47290)
0928 21:12:30.543792 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.543808 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.543963 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.544282 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.544286 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.544307 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.544425 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.544432 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.545335 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.545339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.546618 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.546629 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.547398 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.547401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.547509 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.547533 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.547614 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.547687 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260928 21:12:30.547896 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.543780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47290 (local address 127.0.98.60:43063)
0928 21:12:30.543896 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.543900 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.543914 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:30.544083 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.544086 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.544147 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.544243 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.544545 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.545223 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.546770 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.547270 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.547371 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.547499 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.547623 (+ 124us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.547716 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260928 21:12:30.548540 7313 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260928 21:12:30.548972 7342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4061bb912b55432b8bf8cb174d023b32. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.549115 7342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: This master's current role is: LEARNER
W20260928 21:12:30.659834 7626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:30.660130 7626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:30.660208 7626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:30.664536 7626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:30.664600 7626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:30.664686 7626 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:30.670315 7626 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38533
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44633
--tserver_master_addrs=127.0.98.62:35485,127.0.98.61:34725,127.0.98.60:43063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:30.671499 7626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:30.672569 7626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:30.682377 7632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.682394 7634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:30.682384 7631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:30.682801 7626 server_base.cc:1034] running on GCE node
I20260928 21:12:30.683194 7626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:30.683795 7626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:30.684975 7626 hybrid_clock.cc:648] HybridClock initialized: now 1790629950684939 us; error 43 us; skew 500 ppm
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:30.687851 7626 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 21:12:30.688771 7626 webserver.cc:460] Webserver started at http://127.0.98.3:44633/ using document root <none> and password file <none>
I20260928 21:12:30.689386 7626 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:30.689499 7626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:30.693486 7626 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:30.695508 7641 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.696956 7626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:30.697093 7626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "193c0b53f93243fbb0e499c350b63654"
format_stamp: "Formatted at 2026-09-28 21:12:29 on dist-test-slave-v7xz"
server_key: "97fa1b3a640680d02aebf57c67adf9f8"
server_key_iv: "2a6cfb1403d159efafb099e38f1b3b89"
server_key_version: "encryptionkey@0"
I20260928 21:12:30.697644 7626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:30.713807 7626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:30.717041 7626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:30.717244 7626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:30.717849 7626 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:30.718725 7626 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:30.718796 7626 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.718865 7626 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:30.718914 7626 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:30.728197 7626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38533
I20260928 21:12:30.728222 7754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38533 every 8 connection(s)
I20260928 21:12:30.729272 7626 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:30.738718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 7626
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 28 21:12:30 dist-test-slave-v7xz krb5kdc[6474](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629950, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260928 21:12:30.747280 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.731153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53243 (local address 127.0.98.61:34725)
0928 21:12:30.731280 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.731285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.732451 (+ 1166us) server_negotiation.cc:407] Connection header received
0928 21:12:30.733728 (+ 1277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.733732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.733791 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.733844 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.735723 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.736336 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.737708 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.738004 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.738106 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.742348 (+ 4242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.742352 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.742354 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.742394 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.744170 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.744821 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.744855 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.744858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.744943 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745343 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.745346 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.745347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.745574 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.745663 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.747027 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.747139 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:12:30.747296 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.732936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34781 (local address 127.0.98.60:43063)
0928 21:12:30.733079 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.733088 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.733185 (+ 97us) server_negotiation.cc:407] Connection header received
0928 21:12:30.733708 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.733712 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.733772 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.733926 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.735343 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.736026 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.737677 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.737984 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.738102 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.743165 (+ 5063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.743172 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.743178 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.743207 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.745129 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745702 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.745708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.745713 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.745803 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.746157 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.746162 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.746164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.746371 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.746469 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.747035 (+ 566us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.747148 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:12:30.747711 7759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.731680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:33555)
0928 21:12:30.732278 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.732341 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.733498 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.734103 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.734111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.734572 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.735566 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.735575 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.736472 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.736505 (+ 33us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.737372 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.737382 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.737937 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.737941 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.738132 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.738908 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.738927 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.742622 (+ 3695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.745224 (+ 2602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745227 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.745230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.745566 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.745903 (+ 337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745905 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.745907 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.745992 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.746434 (+ 442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.746437 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.746523 (+ 86us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.746610 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.746710 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":295,"spinlock_wait_cycles":85760,"thread_start_us":139,"threads_started":1}
I20260928 21:12:30.748626 6558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.731153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33555 (local address 127.0.98.62:35485)
0928 21:12:30.731280 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.731285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.732453 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 21:12:30.733812 (+ 1359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.733818 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.733884 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.733954 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.735707 (+ 1753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.736362 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.737498 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.737783 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.737882 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.742764 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.742771 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.742776 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.742803 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.745120 (+ 2317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745693 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.745696 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.745698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.745779 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.746092 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.746095 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.746097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.746306 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.746392 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.748370 (+ 1978us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.748484 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:12:30.748812 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.731680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.98.3:53243)
0928 21:12:30.732278 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.732341 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.733498 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.734103 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.734111 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.734570 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.735572 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.735577 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.736474 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.736478 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.737564 (+ 1086us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.737574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.738239 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.738242 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.738334 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.739578 (+ 1244us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.739589 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.742171 (+ 2582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.744301 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.744307 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.744324 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.744701 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.745053 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745057 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.745059 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.745239 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.745712 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.745719 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.746059 (+ 340us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.746271 (+ 212us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.746635 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":362,"mutex_wait_us":188,"spinlock_wait_cycles":14592,"thread_start_us":155,"threads_started":1}
I20260928 21:12:30.749219 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.732801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.98.3:34781)
0928 21:12:30.733111 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.733126 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.733511 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.734102 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.734109 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.734556 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:30.735182 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.735197 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.736176 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.736179 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.737564 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.737574 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.738105 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.738109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.738199 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.740050 (+ 1851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:30.740060 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:30.743017 (+ 2957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:30.745242 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745245 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.745247 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.745598 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.745927 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:30.745933 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:30.745937 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:30.746042 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:30.746527 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:30.746530 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:30.746623 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.746716 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.746851 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":137,"spinlock_wait_cycles":10112,"thread_start_us":136,"threads_started":1}
I20260928 21:12:30.749881 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35485
I20260928 21:12:30.750020 7757 heartbeater.cc:347] Connected to a master server at 127.0.98.61:34725
I20260928 21:12:30.750121 7756 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.750245 7755 heartbeater.cc:347] Connected to a master server at 127.0.98.60:43063
I20260928 21:12:30.750573 7755 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.750595 7757 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:30.751076 7757 heartbeater.cc:511] Master 127.0.98.61:34725 requested a full tablet report, sending...
I20260928 21:12:30.751077 7755 heartbeater.cc:511] Master 127.0.98.60:43063 requested a full tablet report, sending...
I20260928 21:12:30.751076 7756 heartbeater.cc:511] Master 127.0.98.62:35485 requested a full tablet report, sending...
I20260928 21:12:30.752372 7303 ts_manager.cc:194] Registered new tserver with Master: 193c0b53f93243fbb0e499c350b63654 (127.0.98.3:38533)
I20260928 21:12:30.752583 6519 ts_manager.cc:194] Re-registered known tserver with Master: 193c0b53f93243fbb0e499c350b63654 (127.0.98.3:38533)
I20260928 21:12:30.753409 6519 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33555
I20260928 21:12:30.754616 7119 ts_manager.cc:194] Registered new tserver with Master: 193c0b53f93243fbb0e499c350b63654 (127.0.98.3:38533)
I20260928 21:12:30.757565 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.741084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47292 (local address 127.0.98.60:43063)
0928 21:12:30.741658 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.741663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.742087 (+ 424us) server_negotiation.cc:407] Connection header received
0928 21:12:30.742127 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.742130 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.742172 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.742249 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.744881 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.745670 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.749789 (+ 4119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.750308 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.750388 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.753604 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.753609 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.753611 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.753636 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.755700 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.756121 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.756124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.756126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.756242 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.756558 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.756562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.756563 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.756741 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.756915 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.757247 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.757371 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":81,"threads_started":1}
I20260928 21:12:30.758517 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.740746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33486 (local address 127.0.98.62:35485)
0928 21:12:30.741020 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.741025 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.741331 (+ 306us) server_negotiation.cc:407] Connection header received
0928 21:12:30.741487 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.741490 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.741546 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.741638 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.746733 (+ 5095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.747544 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.751570 (+ 4026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.751901 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.751979 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.754609 (+ 2630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.754616 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.754625 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.754653 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.756329 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.756893 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.756897 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.756899 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.756987 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.757300 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.757306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.757308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.757538 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.757650 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.758226 (+ 576us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.758348 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":106,"threads_started":1}
I20260928 21:12:30.759240 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.740982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39726 (local address 127.0.98.61:34725)
0928 21:12:30.741202 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.741206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.741779 (+ 573us) server_negotiation.cc:407] Connection header received
0928 21:12:30.741824 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.741827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.741873 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.741953 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:30.743250 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.744102 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.751256 (+ 7154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.751623 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.751708 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.753374 (+ 1666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:30.753379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:30.753384 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:30.753414 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:30.757130 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.757615 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.757618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.757620 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.757706 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:30.757996 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:30.757999 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:30.758001 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:30.758222 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:30.758341 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.758924 (+ 583us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.759053 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260928 21:12:30.761237 6519 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33486:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:30.764161 6519 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:30.767483 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: true } } }
I20260928 21:12:30.767627 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.771495 7266 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: attempting to promote NON_VOTER 4da322c2a8994151876290659f8ecb04 to VOTER
I20260928 21:12:30.771883 7266 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:30.773277 7312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 LEARNER]: Refusing update from remote peer 4061bb912b55432b8bf8cb174d023b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 21:12:30.773277 7128 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Refusing update from remote peer 4061bb912b55432b8bf8cb174d023b32: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260928 21:12:30.773644 7773 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 21:12:30.774027 7774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260928 21:12:30.775466 7773 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4da322c2a8994151876290659f8ecb04 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.775870 7128 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4da322c2a8994151876290659f8ecb04 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.777166 7774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.777268 7774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.777215 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.777336 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.777627 7783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.777760 7783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:30.779074 7773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.779186 7773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:30.779541 7312 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 4da322c2a8994151876290659f8ecb04 (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.781824 7342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4061bb912b55432b8bf8cb174d023b32" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:30.781960 7342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:30.784953 7269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.779565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44579 (local address 127.0.0.1:34156)
0928 21:12:30.779740 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.779756 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.779834 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.780705 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.780707 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.780731 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.780816 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.780821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.782306 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.782309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.783390 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.783399 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.784507 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.784510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.784599 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.784623 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.784679 (+ 56us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.784752 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20260928 21:12:30.784992 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.779738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34156 (local address 127.0.98.1:44579)
0928 21:12:30.780177 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.780199 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.780228 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:12:30.780313 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.780319 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.780554 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.780717 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.781021 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.782156 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.783532 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.784379 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.784510 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.784658 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.784717 (+ 59us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.784806 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":112,"threads_started":1}
I20260928 21:12:30.787616 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.781431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38533 (local address 127.0.0.1:59984)
0928 21:12:30.782833 (+ 1402us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.782847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.782934 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.783726 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.783728 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.783747 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.783831 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.783839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.784928 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.784931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.786042 (+ 1111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.786052 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.787194 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.787198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.787290 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.787312 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.787398 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.787476 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1343,"mutex_wait_us":33,"thread_start_us":83,"threads_started":1}
I20260928 21:12:30.787643 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.782362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59984 (local address 127.0.98.3:38533)
0928 21:12:30.783101 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.783116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.783138 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:12:30.783304 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.783310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.783598 (+ 288us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.783753 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.783924 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.784757 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.786181 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.787073 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.787198 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.787340 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.787426 (+ 86us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.787515 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223,"thread_start_us":108,"threads_started":1}
I20260928 21:12:30.788132 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.781960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53846 (local address 127.0.98.2:43489)
0928 21:12:30.782832 (+ 872us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.782846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.782874 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:12:30.782953 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.782962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.783214 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.783382 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.783786 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.784888 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.786330 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.787096 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.787200 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.787852 (+ 652us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.787903 (+ 51us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.788001 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":770,"thread_start_us":115,"threads_started":1}
I20260928 21:12:30.787762 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.781088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43489 (local address 127.0.0.1:53846)
0928 21:12:30.781381 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.781396 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.781482 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.783559 (+ 2077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.783562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.783579 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.783662 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.783667 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.785020 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.785023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.786193 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.786203 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.787386 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.787390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.787487 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.787508 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.787570 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.787636 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":2,"thread_start_us":121,"threads_started":1}
I20260928 21:12:30.788571 7416 tablet_service.cc:1467] Processing CreateTablet for tablet 22988e28cc234393b5070b1a3422424b (DEFAULT_TABLE table=test-table [id=df47c1b2e99f4bcda84302ead925cc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:30.789696 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22988e28cc234393b5070b1a3422424b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:30.790915 7689 tablet_service.cc:1467] Processing CreateTablet for tablet 22988e28cc234393b5070b1a3422424b (DEFAULT_TABLE table=test-table [id=df47c1b2e99f4bcda84302ead925cc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:30.790915 7553 tablet_service.cc:1467] Processing CreateTablet for tablet 22988e28cc234393b5070b1a3422424b (DEFAULT_TABLE table=test-table [id=df47c1b2e99f4bcda84302ead925cc0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:30.791985 7553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22988e28cc234393b5070b1a3422424b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:30.791985 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 22988e28cc234393b5070b1a3422424b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:30.797317 7789 tablet_bootstrap.cc:492] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Bootstrap starting.
I20260928 21:12:30.799440 7789 tablet_bootstrap.cc:654] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:30.800801 7790 tablet_bootstrap.cc:492] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Bootstrap starting.
I20260928 21:12:30.800814 7789 log.cc:824] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:30.801299 7791 tablet_bootstrap.cc:492] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Bootstrap starting.
I20260928 21:12:30.803655 7790 tablet_bootstrap.cc:654] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:30.803874 7791 tablet_bootstrap.cc:654] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:30.804464 7790 log.cc:824] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:30.804574 7791 log.cc:824] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:30.806370 7791 tablet_bootstrap.cc:492] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: No bootstrap required, opened a new log
I20260928 21:12:30.806547 7789 tablet_bootstrap.cc:492] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: No bootstrap required, opened a new log
I20260928 21:12:30.806623 7791 ts_tablet_manager.cc:1397] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:12:30.806742 7789 ts_tablet_manager.cc:1397] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 21:12:30.810388 7790 tablet_bootstrap.cc:492] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: No bootstrap required, opened a new log
I20260928 21:12:30.810689 7790 ts_tablet_manager.cc:1397] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260928 21:12:30.811126 7789 raft_consensus.cc:348] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.811131 7791 raft_consensus.cc:348] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.811386 7789 raft_consensus.cc:374] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:30.811401 7791 raft_consensus.cc:374] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:30.811442 7789 raft_consensus.cc:729] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c51d9d9104357bfa28537aa822e7f, State: Initialized, Role: FOLLOWER
I20260928 21:12:30.811452 7791 raft_consensus.cc:729] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feda8c526d274210beb5977a68ba296d, State: Initialized, Role: FOLLOWER
I20260928 21:12:30.811944 7791 consensus_queue.cc:260] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.811951 7789 consensus_queue.cc:260] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.813019 7789 ts_tablet_manager.cc:1428] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:12:30.813047 7791 ts_tablet_manager.cc:1428] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:12:30.813360 7620 heartbeater.cc:503] Master 127.0.98.62:35485 was elected leader, sending a full tablet report...
I20260928 21:12:30.816808 7790 raft_consensus.cc:348] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.817137 7790 raft_consensus.cc:374] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:30.817196 7790 raft_consensus.cc:729] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193c0b53f93243fbb0e499c350b63654, State: Initialized, Role: FOLLOWER
I20260928 21:12:30.817349 7483 heartbeater.cc:503] Master 127.0.98.62:35485 was elected leader, sending a full tablet report...
I20260928 21:12:30.817644 7790 consensus_queue.cc:260] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.818747 7790 ts_tablet_manager.cc:1428] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 21:12:30.819303 7756 heartbeater.cc:503] Master 127.0.98.62:35485 was elected leader, sending a full tablet report...
I20260928 21:12:30.937276 7797 raft_consensus.cc:482] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:30.937490 7797 raft_consensus.cc:504] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.938628 7797 leader_election.cc:288] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579), feda8c526d274210beb5977a68ba296d (127.0.98.2:43489)
I20260928 21:12:30.942625 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.939132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54017 (local address 127.0.98.1:44579)
0928 21:12:30.939259 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.939264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.939281 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:12:30.939347 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.939350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.939404 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.939508 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.939812 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.940363 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.941598 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.941984 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.942109 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.942175 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.942400 (+ 225us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.942484 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:12:30.942628 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.939054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40621 (local address 127.0.98.2:43489)
0928 21:12:30.939194 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.939199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.939368 (+ 169us) server_negotiation.cc:407] Connection header received
0928 21:12:30.939541 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.939544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.939601 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.939678 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:30.939956 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.940464 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.941573 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.941981 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.942111 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.942174 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.942400 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.942483 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:12:30.942655 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.938899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44579 (local address 127.0.98.3:54017)
0928 21:12:30.939054 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.939075 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.939214 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.939531 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.939534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.939552 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.939691 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.939698 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.940488 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.940492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.941445 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.941455 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.942123 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.942126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.942266 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.942293 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.942382 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.942497 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":79}
I20260928 21:12:30.942662 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.938899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43489 (local address 127.0.98.3:40621)
0928 21:12:30.939280 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:30.939294 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:30.939422 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:30.939699 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:30.939701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:30.939716 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:30.939844 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.939850 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.940581 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.940584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.941434 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:30.941443 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.942112 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.942116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:30.942268 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.942291 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:12:30.942382 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:12:30.942500 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":313}
I20260928 21:12:30.943467 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22988e28cc234393b5070b1a3422424b" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feda8c526d274210beb5977a68ba296d" is_pre_election: true
I20260928 21:12:30.943493 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22988e28cc234393b5070b1a3422424b" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c51d9d9104357bfa28537aa822e7f" is_pre_election: true
I20260928 21:12:30.943773 7436 raft_consensus.cc:2454] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 0.
I20260928 21:12:30.943789 7573 raft_consensus.cc:2454] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 0.
I20260928 21:12:30.944298 7643 leader_election.cc:302] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 193c0b53f93243fbb0e499c350b63654, f53c51d9d9104357bfa28537aa822e7f, feda8c526d274210beb5977a68ba296d; no voters:
I20260928 21:12:30.944618 7797 raft_consensus.cc:2781] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:30.944693 7797 raft_consensus.cc:482] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:30.944725 7797 raft_consensus.cc:3037] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:30.945852 7797 raft_consensus.cc:504] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.946269 7797 leader_election.cc:288] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 election: Requested vote from peers f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579), feda8c526d274210beb5977a68ba296d (127.0.98.2:43489)
I20260928 21:12:30.946635 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22988e28cc234393b5070b1a3422424b" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c51d9d9104357bfa28537aa822e7f"
I20260928 21:12:30.946727 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "22988e28cc234393b5070b1a3422424b" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feda8c526d274210beb5977a68ba296d"
I20260928 21:12:30.946787 7436 raft_consensus.cc:3037] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:30.946866 7573 raft_consensus.cc:3037] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:30.947849 7436 raft_consensus.cc:2454] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 1.
I20260928 21:12:30.947939 7573 raft_consensus.cc:2454] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 1.
I20260928 21:12:30.948215 7645 leader_election.cc:302] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193c0b53f93243fbb0e499c350b63654, f53c51d9d9104357bfa28537aa822e7f; no voters:
I20260928 21:12:30.948410 7797 raft_consensus.cc:2781] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:30.948618 7797 raft_consensus.cc:686] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 1 LEADER]: Becoming Leader. State: Replica: 193c0b53f93243fbb0e499c350b63654, State: Running, Role: LEADER
I20260928 21:12:30.949108 7797 consensus_queue.cc:237] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } }
I20260928 21:12:30.952615 6518 catalog_manager.cc:5495] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 193c0b53f93243fbb0e499c350b63654 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "193c0b53f93243fbb0e499c350b63654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } health_report { overall_health: HEALTHY } } }
I20260928 21:12:30.974633 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:30.970930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59986 (local address 127.0.98.3:38533)
0928 21:12:30.971062 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:30.971066 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:30.971083 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:12:30.971296 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:30.971298 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:30.971353 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:30.971508 (+ 155us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:30.971774 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.972271 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.973169 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:30.973385 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:30.973468 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:30.973815 (+ 347us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:30.974237 (+ 422us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:30.974327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:30.974460 (+ 133us) server_negotiation.cc:299] Negotiation successful
0928 21:12:30.974510 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260928 21:12:30.980933 7758 tablet.cc:2369] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:30.981172 7436 raft_consensus.cc:1261] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 1 FOLLOWER]: Refusing update from remote peer 193c0b53f93243fbb0e499c350b63654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:30.981179 7573 raft_consensus.cc:1261] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Refusing update from remote peer 193c0b53f93243fbb0e499c350b63654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:30.981973 7798 consensus_queue.cc:1035] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:30.982231 7797 consensus_queue.cc:1035] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:30.991057 7801 mvcc.cc:205] Tried to move back new op lower bound from 7334420279210156032 to 7334420279090135040. Current Snapshot: MvccSnapshot[applied={T|T < 7334420279210156032}]
I20260928 21:12:30.991117 7802 mvcc.cc:205] Tried to move back new op lower bound from 7334420279210156032 to 7334420279090135040. Current Snapshot: MvccSnapshot[applied={T|T < 7334420279210156032}]
I20260928 21:12:30.996577 7803 mvcc.cc:205] Tried to move back new op lower bound from 7334420279210156032 to 7334420279090135040. Current Snapshot: MvccSnapshot[applied={T|T < 7334420279210156032 or (T in {7334420279210156032})}]
I20260928 21:12:31.002553 6518 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:31.002743 6518 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33486:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:31.007764 6518 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4061bb912b55432b8bf8cb174d023b32: Sending DeleteTablet for 3 replicas of tablet 22988e28cc234393b5070b1a3422424b
I20260928 21:12:31.008477 7416 tablet_service.cc:1514] Processing DeleteTablet for tablet 22988e28cc234393b5070b1a3422424b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34156
I20260928 21:12:31.008908 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet 22988e28cc234393b5070b1a3422424b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59984
I20260928 21:12:31.009042 7553 tablet_service.cc:1514] Processing DeleteTablet for tablet 22988e28cc234393b5070b1a3422424b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:30 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53846
I20260928 21:12:31.009356 7811 tablet_replica.cc:330] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: stopping tablet replica
I20260928 21:12:31.009541 7812 tablet_replica.cc:330] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: stopping tablet replica
I20260928 21:12:31.009702 7811 raft_consensus.cc:2229] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:31.009805 7812 raft_consensus.cc:2229] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:31.010077 7811 raft_consensus.cc:2258] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:31.010128 7812 raft_consensus.cc:2258] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:31.011922 7811 ts_tablet_manager.cc:1905] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:31.012301 7812 ts_tablet_manager.cc:1905] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:31.012730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6487
I20260928 21:12:31.015777 7811 ts_tablet_manager.cc:1918] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:31.015894 7811 log.cc:1196] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/22988e28cc234393b5070b1a3422424b
I20260928 21:12:31.015884 7812 ts_tablet_manager.cc:1918] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:31.015969 7812 log.cc:1196] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/22988e28cc234393b5070b1a3422424b
I20260928 21:12:31.016266 7811 ts_tablet_manager.cc:1939] T 22988e28cc234393b5070b1a3422424b P 193c0b53f93243fbb0e499c350b63654: Deleting consensus metadata
I20260928 21:12:31.016309 7812 ts_tablet_manager.cc:1939] T 22988e28cc234393b5070b1a3422424b P feda8c526d274210beb5977a68ba296d: Deleting consensus metadata
W20260928 21:12:31.020404 7644 connection.cc:535] server connection from 127.0.0.1:59984 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260928 21:12:31.020418 7509 connection.cc:535] server connection from 127.0.0.1:53846 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:12:31.021786 7810 tablet_replica.cc:330] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: stopping tablet replica
I20260928 21:12:31.022147 7810 raft_consensus.cc:2229] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:31.022516 7810 raft_consensus.cc:2258] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:31.024518 7810 ts_tablet_manager.cc:1905] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:31.027864 7810 ts_tablet_manager.cc:1918] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:31.027976 7810 log.cc:1196] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/22988e28cc234393b5070b1a3422424b
I20260928 21:12:31.028311 7810 ts_tablet_manager.cc:1939] T 22988e28cc234393b5070b1a3422424b P f53c51d9d9104357bfa28537aa822e7f: Deleting consensus metadata
W20260928 21:12:31.029482 7372 connection.cc:441] server connection from 127.0.0.1:34156 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34156 (request call id 1) could send its response
I20260928 21:12:31.031693 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.011351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39730 (local address 127.0.98.61:34725)
0928 21:12:31.011472 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:31.011477 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:31.011805 (+ 328us) server_negotiation.cc:407] Connection header received
0928 21:12:31.011989 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:31.011992 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:31.012040 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:31.012103 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:31.021771 (+ 9668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.022704 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.024696 (+ 1992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.025128 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.025207 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:31.027375 (+ 2168us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:31.027380 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:31.027383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:31.027411 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:31.029443 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.029970 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.029974 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.029976 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.030063 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.030375 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.030378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.030380 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.030885 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:31.031000 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:31.031375 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 21:12:31.031507 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:12:31.035934 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.024387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47318 (local address 127.0.98.60:43063)
0928 21:12:31.024507 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:31.024512 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:31.025444 (+ 932us) server_negotiation.cc:407] Connection header received
0928 21:12:31.026603 (+ 1159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:31.026609 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:31.026667 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:31.026725 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:31.027866 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.028678 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.029781 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.030105 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.030180 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:31.031778 (+ 1598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:31.031782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:31.031785 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:31.031811 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:31.033835 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.034361 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.034368 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.034371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.034466 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.034810 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.034816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.034823 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.035084 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:31.035206 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:31.035610 (+ 404us) server_negotiation.cc:299] Negotiation successful
0928 21:12:31.035736 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:12:31.038015 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.023853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39734 (local address 127.0.98.61:34725)
0928 21:12:31.023980 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:31.023985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:31.025257 (+ 1272us) server_negotiation.cc:407] Connection header received
0928 21:12:31.025618 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:31.025621 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:31.025664 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:31.025712 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:31.029844 (+ 4132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.030542 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.031775 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.032125 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.032202 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:31.034018 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:31.034023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:31.034025 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:31.034051 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:31.035753 (+ 1702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.036223 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.036228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.036232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.036317 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.037033 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.037036 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.037039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.037262 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:31.037372 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:31.037726 (+ 354us) server_negotiation.cc:299] Negotiation successful
0928 21:12:31.037849 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:12:31.043330 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.011423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47302 (local address 127.0.98.60:43063)
0928 21:12:31.011531 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:31.011535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:31.033979 (+ 22444us) server_negotiation.cc:407] Connection header received
0928 21:12:31.034153 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:31.034156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:31.034206 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:31.034259 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:31.035046 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.035932 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.037276 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:31.037613 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:31.037705 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:31.039582 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:31.039586 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:31.039590 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:31.039616 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:31.041546 (+ 1930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.041939 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.041942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.041944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.042035 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:31.042334 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:31.042338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:31.042340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:31.042569 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:31.042672 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:31.043022 (+ 350us) server_negotiation.cc:299] Negotiation successful
0928 21:12:31.043158 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:12:31.995958 7821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.995429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:44595)
0928 21:12:31.995716 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:31.995820 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":78,"threads_started":1}
W20260928 21:12:31.996343 7483 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:31.997380 7822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.996698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:59691)
0928 21:12:31.996961 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:31.997285 (+ 324us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":76,"threads_started":1}
W20260928 21:12:31.997656 7620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:31.997810 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:31.997366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:57003)
0928 21:12:31.997631 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:31.997716 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":65,"threads_started":1}
W20260928 21:12:31.998055 7756 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:35485 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:32.508201 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4061bb912b55432b8bf8cb174d023b32)
I20260928 21:12:32.508478 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:32.508912 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4061bb912b55432b8bf8cb174d023b32)
I20260928 21:12:32.509112 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:32.509730 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485), 4da322c2a8994151876290659f8ecb04 (127.0.98.60:43063)
I20260928 21:12:32.509994 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485), 4b16a1c234244e14b558d76fc5958e29 (127.0.98.61:34725)
I20260928 21:12:32.510416 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.509892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33942)
0928 21:12:32.510225 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.510311 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":77,"threads_started":1}
I20260928 21:12:32.510697 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.510069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33948)
0928 21:12:32.510451 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.510611 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":64,"threads_started":1}
W20260928 21:12:32.510737 7104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
W20260928 21:12:32.510948 7288 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:32.511613 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.511380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33958)
0928 21:12:32.511501 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.511552 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 21:12:32.511790 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:32.512130 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.511826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33970)
0928 21:12:32.512025 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.512072 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
W20260928 21:12:32.512302 7288 leader_election.cc:334] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:32.514447 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.510333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.0.98.60:43063)
0928 21:12:32.510593 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:32.510597 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:32.510932 (+ 335us) server_negotiation.cc:407] Connection header received
0928 21:12:32.510980 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:32.510983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:32.511031 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:32.511104 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:32.511496 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.512281 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.513490 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.513896 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.514008 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:32.514069 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:32.514265 (+ 196us) server_negotiation.cc:299] Negotiation successful
0928 21:12:32.514329 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":202,"thread_start_us":61,"threads_started":1}
I20260928 21:12:32.514446 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.510479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:43063 (local address 127.0.0.1:47332)
0928 21:12:32.510708 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.510723 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:32.510831 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:32.511236 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:32.511241 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:32.511264 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:32.511370 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:32.511377 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.512415 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.512418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:32.513361 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:32.513369 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.514009 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.514013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:32.514147 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:32.514164 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:12:32.514251 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:12:32.514318 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":22,"thread_start_us":51,"threads_started":1}
I20260928 21:12:32.514887 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.510200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34725 (local address 127.0.0.1:39740)
0928 21:12:32.510687 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.510702 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:32.510805 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:32.511312 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:32.511318 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:32.511353 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:32.511446 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:32.511453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.512522 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.512527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:32.513607 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:32.513635 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.514301 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.514304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:32.514395 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:32.514415 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:12:32.514669 (+ 254us) client_negotiation.cc:240] Negotiation successful
0928 21:12:32.514745 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":55,"threads_started":1}
I20260928 21:12:32.514911 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.510477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39740 (local address 127.0.98.61:34725)
0928 21:12:32.511005 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:32.511011 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:32.511024 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:32.511065 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:32.511068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:32.511114 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:32.511195 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:32.511576 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.512424 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.513731 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:32.514185 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:32.514301 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:32.514363 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:32.514703 (+ 340us) server_negotiation.cc:299] Negotiation successful
0928 21:12:32.514781 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":452,"thread_start_us":53,"threads_started":1}
I20260928 21:12:32.514947 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b16a1c234244e14b558d76fc5958e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4da322c2a8994151876290659f8ecb04" is_pre_election: true
I20260928 21:12:32.515118 7312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b16a1c234244e14b558d76fc5958e29 in term 1.
I20260928 21:12:32.515434 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b16a1c234244e14b558d76fc5958e29, 4da322c2a8994151876290659f8ecb04; no voters: 4061bb912b55432b8bf8cb174d023b32
I20260928 21:12:32.515458 7128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4da322c2a8994151876290659f8ecb04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b16a1c234244e14b558d76fc5958e29" is_pre_election: true
I20260928 21:12:32.515607 7128 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4da322c2a8994151876290659f8ecb04 in term 1.
I20260928 21:12:32.515697 7824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:12:32.515770 7824 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4061bb912b55432b8bf8cb174d023b32)
I20260928 21:12:32.515828 7824 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:12:32.516985 7287 leader_election.cc:302] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4b16a1c234244e14b558d76fc5958e29, 4da322c2a8994151876290659f8ecb04; no voters: 4061bb912b55432b8bf8cb174d023b32
I20260928 21:12:32.517105 7824 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:32.517181 7825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:12:32.517227 7825 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4061bb912b55432b8bf8cb174d023b32)
I20260928 21:12:32.517264 7825 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:12:32.517431 7824 leader_election.cc:288] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 2 election: Requested vote from peers 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485), 4da322c2a8994151876290659f8ecb04 (127.0.98.60:43063)
I20260928 21:12:32.517773 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.517555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33976)
0928 21:12:32.517665 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.517718 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 21:12:32.518332 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.518132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33984)
0928 21:12:32.518234 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.518281 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:12:32.518450 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:32.518544 7825 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:32.518843 7825 leader_election.cc:288] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [CANDIDATE]: Term 2 election: Requested vote from peers 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485), 4b16a1c234244e14b558d76fc5958e29 (127.0.98.61:34725)
I20260928 21:12:32.519007 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b16a1c234244e14b558d76fc5958e29" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4da322c2a8994151876290659f8ecb04"
I20260928 21:12:32.519163 7312 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4b16a1c234244e14b558d76fc5958e29 in current term 2: Already voted for candidate 4da322c2a8994151876290659f8ecb04 in this term.
I20260928 21:12:32.519248 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.519036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:33990)
0928 21:12:32.519149 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.519198 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:12:32.519356 7128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4da322c2a8994151876290659f8ecb04" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4b16a1c234244e14b558d76fc5958e29"
I20260928 21:12:32.519505 7128 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4da322c2a8994151876290659f8ecb04 in current term 2: Already voted for candidate 4b16a1c234244e14b558d76fc5958e29 in this term.
I20260928 21:12:32.519505 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4b16a1c234244e14b558d76fc5958e29; no voters: 4061bb912b55432b8bf8cb174d023b32, 4da322c2a8994151876290659f8ecb04
I20260928 21:12:32.519752 7824 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:12:32.519933 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.519704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:34000)
0928 21:12:32.519829 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.519877 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20260928 21:12:32.520040 7288 leader_election.cc:334] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:32.520102 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4da322c2a8994151876290659f8ecb04; no voters: 4061bb912b55432b8bf8cb174d023b32, 4b16a1c234244e14b558d76fc5958e29
I20260928 21:12:32.520215 7825 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:12:32.997368 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.996913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:39277)
0928 21:12:32.997191 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.997265 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":81,"threads_started":1}
I20260928 21:12:32.998584 7835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.998151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:41973)
0928 21:12:32.998438 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.998495 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":76,"threads_started":1}
I20260928 21:12:32.998924 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:32.998499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:37957)
0928 21:12:32.998774 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:32.998835 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":61,"threads_started":1}
I20260928 21:12:33.998509 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:33.998020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:39069)
0928 21:12:33.998312 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:33.998384 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":95,"threads_started":1}
I20260928 21:12:33.999614 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:33.999215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:35887)
0928 21:12:33.999442 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:33.999514 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":68,"threads_started":1}
I20260928 21:12:33.999922 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:33.999531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:56827)
0928 21:12:33.999749 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:33.999812 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":75,"threads_started":1}
I20260928 21:12:34.515760 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:34.515893 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:34.516381 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485), 4da322c2a8994151876290659f8ecb04 (127.0.98.60:43063)
I20260928 21:12:34.516999 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.516565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:51704)
0928 21:12:34.516799 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.516905 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":58,"threads_started":1}
I20260928 21:12:34.517161 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b16a1c234244e14b558d76fc5958e29" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4da322c2a8994151876290659f8ecb04" is_pre_election: true
I20260928 21:12:34.517326 7312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b16a1c234244e14b558d76fc5958e29 in term 2.
I20260928 21:12:34.517668 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b16a1c234244e14b558d76fc5958e29, 4da322c2a8994151876290659f8ecb04; no voters:
I20260928 21:12:34.517864 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260928 21:12:34.517933 7840 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:34.517972 7840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:12:34.518209 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.517995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:51712)
0928 21:12:34.518104 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.518156 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260928 21:12:34.518368 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:34.519210 7840 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:34.519536 7840 leader_election.cc:288] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 3 election: Requested vote from peers 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485), 4da322c2a8994151876290659f8ecb04 (127.0.98.60:43063)
I20260928 21:12:34.519910 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.519697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:51716)
0928 21:12:34.519792 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.519843 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260928 21:12:34.520078 7312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4b16a1c234244e14b558d76fc5958e29" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4da322c2a8994151876290659f8ecb04"
I20260928 21:12:34.520195 7312 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:12:34.521175 7312 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b16a1c234244e14b558d76fc5958e29 in term 3.
I20260928 21:12:34.521490 7104 leader_election.cc:302] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b16a1c234244e14b558d76fc5958e29, 4da322c2a8994151876290659f8ecb04; no voters:
I20260928 21:12:34.521579 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.521317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:51718)
0928 21:12:34.521433 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.521477 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:12:34.521652 7104 leader_election.cc:334] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
I20260928 21:12:34.521673 7840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:12:34.521833 7840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [term 3 LEADER]: Becoming Leader. State: Replica: 4b16a1c234244e14b558d76fc5958e29, State: Running, Role: LEADER
I20260928 21:12:34.522135 7840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } }
I20260928 21:12:34.523651 7843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b16a1c234244e14b558d76fc5958e29. Latest consensus state: current_term: 3 leader_uuid: "4b16a1c234244e14b558d76fc5958e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:34.523751 7843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:34.524051 7845 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:34.526288 7845 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:34.526648 7845 catalog_manager.cc:1259] Loaded cluster ID: 143d04b7ee5c4aa78770c9adffcc3f08
I20260928 21:12:34.526690 7845 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:34.527190 7845 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:34.527530 7845 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Loaded TSK: 0
I20260928 21:12:34.528170 7845 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:34.591770 7118 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39734:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:34.594827 7118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:34.598141 7312 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [term 3 FOLLOWER]: Refusing update from remote peer 4b16a1c234244e14b558d76fc5958e29: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260928 21:12:34.598183 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.597874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:51732)
0928 21:12:34.598010 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.598072 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 21:12:34.598647 7843 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260928 21:12:34.598923 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.598692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.0.1:51748)
0928 21:12:34.598813 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.598865 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 21:12:34.599081 7104 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 -> Peer 4061bb912b55432b8bf8cb174d023b32 (127.0.98.62:35485): Couldn't send request to peer 4061bb912b55432b8bf8cb174d023b32. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260928 21:12:34.600353 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b16a1c234244e14b558d76fc5958e29. Latest consensus state: current_term: 3 leader_uuid: "4b16a1c234244e14b558d76fc5958e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:34.600468 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:34.601605 7840 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4b16a1c234244e14b558d76fc5958e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:34.601852 7848 mvcc.cc:205] Tried to move back new op lower bound from 7334420294027902976 to 7334420293725437952. Current Snapshot: MvccSnapshot[applied={T|T < 7334420294027902976}]
I20260928 21:12:34.602267 7847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4b16a1c234244e14b558d76fc5958e29" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "4061bb912b55432b8bf8cb174d023b32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35485 } } peers { permanent_uuid: "4b16a1c234244e14b558d76fc5958e29" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34725 } attrs { promote: false } } peers { permanent_uuid: "4da322c2a8994151876290659f8ecb04" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 43063 } attrs { promote: false } } }
I20260928 21:12:34.602401 7847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da322c2a8994151876290659f8ecb04 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:12:34.603972 7840 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:34.611913 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.607463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50070 (local address 127.0.98.3:38533)
0928 21:12:34.607756 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:34.607761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:34.608042 (+ 281us) server_negotiation.cc:407] Connection header received
0928 21:12:34.608102 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:34.608117 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:34.608168 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:34.608264 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:34.608542 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.609365 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.610680 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.611348 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.611617 (+ 269us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.611680 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:34.611719 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:12:34.611784 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":67,"threads_started":1}
I20260928 21:12:34.611912 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.607354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38533 (local address 127.0.0.1:50070)
0928 21:12:34.607442 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.607459 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:34.607543 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:34.608280 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:34.608282 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:34.608294 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:34.608418 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:34.608424 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.609447 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.609453 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:34.610549 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:34.610556 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.611484 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.611488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:34.611618 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.611640 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:12:34.611704 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:12:34.611774 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":95}
I20260928 21:12:34.612113 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.607877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43314 (local address 127.0.98.2:43489)
0928 21:12:34.608168 (+ 291us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:34.608173 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:34.608187 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:34.608241 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:34.608246 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:34.608292 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:34.608369 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:34.608618 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.609387 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.610756 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.611346 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.611463 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.611531 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:34.611874 (+ 343us) server_negotiation.cc:299] Negotiation successful
0928 21:12:34.611954 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":68,"threads_started":1}
I20260928 21:12:34.612201 7854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.607544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43489 (local address 127.0.0.1:43314)
0928 21:12:34.607961 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.607974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:34.608056 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:34.608384 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:34.608387 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:34.608399 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:34.608501 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:34.608507 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.609509 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.609513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:34.610658 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:34.610667 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.611460 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.611463 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:34.611552 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.611601 (+ 49us) client_negotiation.cc:769] Sending connection context
0928 21:12:34.611667 (+ 66us) client_negotiation.cc:240] Negotiation successful
0928 21:12:34.611728 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":15,"thread_start_us":101,"threads_started":1}
I20260928 21:12:34.612615 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.607829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44579 (local address 127.0.0.1:37298)
0928 21:12:34.608806 (+ 977us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.608818 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:34.608940 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:34.609194 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:34.609196 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:34.609211 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:34.609299 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:34.609304 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.610282 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.610285 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:34.611209 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:34.611216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.612238 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.612243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:34.612329 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.612349 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:12:34.612419 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:12:34.612509 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"thread_start_us":99,"threads_started":1}
I20260928 21:12:34.612892 7553 tablet_service.cc:1467] Processing CreateTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 (DEFAULT_TABLE table=test-table [id=18e185ca6bfa4580aff284fe37d34b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:34.613042 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.607913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37298 (local address 127.0.98.1:44579)
0928 21:12:34.608682 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:34.608687 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:34.608915 (+ 228us) server_negotiation.cc:407] Connection header received
0928 21:12:34.609035 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:34.609038 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:34.609084 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:34.609168 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:34.609418 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.610160 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.611320 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.611968 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.612059 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.612178 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:34.612775 (+ 597us) server_negotiation.cc:299] Negotiation successful
0928 21:12:34.612874 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":62,"threads_started":1}
I20260928 21:12:34.613211 7553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad3e1d7edfa444d96ef1a219c2950a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:34.615302 7858 tablet_bootstrap.cc:492] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Bootstrap starting.
I20260928 21:12:34.615778 7416 tablet_service.cc:1467] Processing CreateTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 (DEFAULT_TABLE table=test-table [id=18e185ca6bfa4580aff284fe37d34b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:34.616043 7416 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad3e1d7edfa444d96ef1a219c2950a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:34.615999 7689 tablet_service.cc:1467] Processing CreateTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 (DEFAULT_TABLE table=test-table [id=18e185ca6bfa4580aff284fe37d34b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:34.616228 7858 tablet_bootstrap.cc:654] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:34.616268 7689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fad3e1d7edfa444d96ef1a219c2950a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:34.617570 7858 tablet_bootstrap.cc:492] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: No bootstrap required, opened a new log
I20260928 21:12:34.617668 7858 ts_tablet_manager.cc:1397] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:34.618130 7858 raft_consensus.cc:348] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.618252 7858 raft_consensus.cc:374] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:34.618288 7858 raft_consensus.cc:729] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: feda8c526d274210beb5977a68ba296d, State: Initialized, Role: FOLLOWER
I20260928 21:12:34.618438 7860 tablet_bootstrap.cc:492] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f: Bootstrap starting.
I20260928 21:12:34.618469 7858 consensus_queue.cc:260] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.618831 7858 ts_tablet_manager.cc:1428] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260928 21:12:34.619153 7862 tablet_bootstrap.cc:492] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Bootstrap starting.
I20260928 21:12:34.620064 7862 tablet_bootstrap.cc:654] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:34.620187 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.619708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:50539)
0928 21:12:34.619964 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.620023 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":103,"threads_started":1}
I20260928 21:12:34.621268 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.621011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:54067)
0928 21:12:34.621115 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.621168 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260928 21:12:34.621680 7621 heartbeater.cc:503] Master 127.0.98.61:34725 was elected leader, sending a full tablet report...
I20260928 21:12:34.621697 7860 tablet_bootstrap.cc:654] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:34.623059 7862 tablet_bootstrap.cc:492] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: No bootstrap required, opened a new log
I20260928 21:12:34.623059 7860 tablet_bootstrap.cc:492] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f: No bootstrap required, opened a new log
I20260928 21:12:34.623153 7862 ts_tablet_manager.cc:1397] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260928 21:12:34.623149 7860 ts_tablet_manager.cc:1397] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 21:12:34.623463 7862 raft_consensus.cc:348] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.623538 7862 raft_consensus.cc:374] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:34.623502 7860 raft_consensus.cc:348] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.623559 7862 raft_consensus.cc:729] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 193c0b53f93243fbb0e499c350b63654, State: Initialized, Role: FOLLOWER
I20260928 21:12:34.623577 7860 raft_consensus.cc:374] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:34.623598 7860 raft_consensus.cc:729] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f53c51d9d9104357bfa28537aa822e7f, State: Initialized, Role: FOLLOWER
I20260928 21:12:34.623651 7862 consensus_queue.cc:260] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.623754 7860 consensus_queue.cc:260] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.623976 7862 ts_tablet_manager.cc:1428] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:34.624056 7860 ts_tablet_manager.cc:1428] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:34.625056 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.624582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:56813)
0928 21:12:34.624823 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.624951 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":97,"threads_started":1}
I20260928 21:12:34.625412 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.624944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:34193)
0928 21:12:34.625259 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.625312 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":111,"threads_started":1}
I20260928 21:12:34.626835 7484 heartbeater.cc:503] Master 127.0.98.61:34725 was elected leader, sending a full tablet report...
I20260928 21:12:34.627442 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.627227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:55663)
0928 21:12:34.627332 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.627383 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:12:34.694062 7866 raft_consensus.cc:482] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:34.694232 7866 raft_consensus.cc:504] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.694700 7866 leader_election.cc:288] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers feda8c526d274210beb5977a68ba296d (127.0.98.2:43489), f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579)
I20260928 21:12:34.695087 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad3e1d7edfa444d96ef1a219c2950a8" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feda8c526d274210beb5977a68ba296d" is_pre_election: true
I20260928 21:12:34.695216 7573 raft_consensus.cc:2454] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 0.
I20260928 21:12:34.695169 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad3e1d7edfa444d96ef1a219c2950a8" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c51d9d9104357bfa28537aa822e7f" is_pre_election: true
I20260928 21:12:34.695358 7436 raft_consensus.cc:2454] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 0.
I20260928 21:12:34.695593 7643 leader_election.cc:302] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193c0b53f93243fbb0e499c350b63654, feda8c526d274210beb5977a68ba296d; no voters:
I20260928 21:12:34.695771 7866 raft_consensus.cc:2781] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:34.695837 7866 raft_consensus.cc:482] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:34.695865 7866 raft_consensus.cc:3037] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:34.697013 7866 raft_consensus.cc:504] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.697367 7866 leader_election.cc:288] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 election: Requested vote from peers feda8c526d274210beb5977a68ba296d (127.0.98.2:43489), f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579)
I20260928 21:12:34.697752 7436 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad3e1d7edfa444d96ef1a219c2950a8" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f53c51d9d9104357bfa28537aa822e7f"
I20260928 21:12:34.697865 7436 raft_consensus.cc:3037] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:34.697813 7573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fad3e1d7edfa444d96ef1a219c2950a8" candidate_uuid: "193c0b53f93243fbb0e499c350b63654" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "feda8c526d274210beb5977a68ba296d"
I20260928 21:12:34.698025 7573 raft_consensus.cc:3037] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:34.698829 7436 raft_consensus.cc:2454] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 1.
I20260928 21:12:34.698982 7573 raft_consensus.cc:2454] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 193c0b53f93243fbb0e499c350b63654 in term 1.
I20260928 21:12:34.699168 7645 leader_election.cc:302] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 193c0b53f93243fbb0e499c350b63654, f53c51d9d9104357bfa28537aa822e7f; no voters:
I20260928 21:12:34.699340 7866 raft_consensus.cc:2781] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:34.699579 7866 raft_consensus.cc:686] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 1 LEADER]: Becoming Leader. State: Replica: 193c0b53f93243fbb0e499c350b63654, State: Running, Role: LEADER
I20260928 21:12:34.699753 7866 consensus_queue.cc:237] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } }
I20260928 21:12:34.699810 7757 heartbeater.cc:503] Master 127.0.98.61:34725 was elected leader, sending a full tablet report...
I20260928 21:12:34.700380 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.700065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:41863)
0928 21:12:34.700185 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.700301 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:12:34.701530 7118 catalog_manager.cc:5495] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 reported cstate change: term changed from 0 to 1, leader changed from <none> to 193c0b53f93243fbb0e499c350b63654 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "193c0b53f93243fbb0e499c350b63654" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "193c0b53f93243fbb0e499c350b63654" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38533 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:34.719611 7853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.716534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50084 (local address 127.0.98.3:38533)
0928 21:12:34.716658 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:34.716662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:34.716675 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:34.716725 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:34.716727 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:34.716772 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:34.716900 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:34.717186 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.717690 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.718579 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:34.718794 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:34.718883 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:34.719169 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:34.719236 (+ 67us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:34.719294 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:34.719452 (+ 158us) server_negotiation.cc:299] Negotiation successful
0928 21:12:34.719501 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:12:34.721976 7573 raft_consensus.cc:1261] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Refusing update from remote peer 193c0b53f93243fbb0e499c350b63654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:34.722002 7436 raft_consensus.cc:1261] T fad3e1d7edfa444d96ef1a219c2950a8 P f53c51d9d9104357bfa28537aa822e7f [term 1 FOLLOWER]: Refusing update from remote peer 193c0b53f93243fbb0e499c350b63654: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:34.722468 7866 consensus_queue.cc:1035] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "feda8c526d274210beb5977a68ba296d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:34.722685 7870 consensus_queue.cc:1035] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f53c51d9d9104357bfa28537aa822e7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:34.724643 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.724030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:46997)
0928 21:12:34.724484 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.724564 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377}
I20260928 21:12:34.725292 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.725054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:38073)
0928 21:12:34.725168 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.725223 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260928 21:12:34.727663 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.727383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:56179)
0928 21:12:34.727529 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.727588 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260928 21:12:34.727844 7863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.727425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.2:35231)
0928 21:12:34.727733 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.727782 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":244}
I20260928 21:12:34.731171 7119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:34.731305 7119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39734:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:34.731719 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.731439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.1:32783)
0928 21:12:34.731579 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.731641 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260928 21:12:34.733589 7868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.733343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35485 (local address 127.0.98.3:34693)
0928 21:12:34.733456 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.733519 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:35485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260928 21:12:34.736042 7119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b16a1c234244e14b558d76fc5958e29: Sending DeleteTablet for 3 replicas of tablet fad3e1d7edfa444d96ef1a219c2950a8
I20260928 21:12:34.736666 7689 tablet_service.cc:1514] Processing DeleteTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50070
I20260928 21:12:34.736749 7688 tablet_service.cc:1514] Processing DeleteTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:50070
I20260928 21:12:34.736949 7416 tablet_service.cc:1514] Processing DeleteTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:37298
I20260928 21:12:34.736950 7553 tablet_service.cc:1514] Processing DeleteTablet for tablet fad3e1d7edfa444d96ef1a219c2950a8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:34 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:43314
I20260928 21:12:34.737473 7882 tablet_replica.cc:330] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: stopping tablet replica
I20260928 21:12:34.737596 7882 raft_consensus.cc:2229] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:34.737728 7882 raft_consensus.cc:2258] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:34.737761 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 7263
I20260928 21:12:34.740165 7882 ts_tablet_manager.cc:1905] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:34.743739 7882 ts_tablet_manager.cc:1918] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:34.743844 7882 log.cc:1196] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/fad3e1d7edfa444d96ef1a219c2950a8
I20260928 21:12:34.744161 7882 ts_tablet_manager.cc:1939] T fad3e1d7edfa444d96ef1a219c2950a8 P feda8c526d274210beb5977a68ba296d: Deleting consensus metadata
I20260928 21:12:34.744375 7881 tablet_replica.cc:330] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: stopping tablet replica
I20260928 21:12:34.744566 7881 raft_consensus.cc:2229] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:34.744802 7881 raft_consensus.cc:2258] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:34.745429 7104 catalog_manager.cc:4841] TS feda8c526d274210beb5977a68ba296d (127.0.98.2:43489): tablet fad3e1d7edfa444d96ef1a219c2950a8 (table test-table [id=18e185ca6bfa4580aff284fe37d34b9b]) successfully deleted
I20260928 21:12:34.745523 7881 ts_tablet_manager.cc:1905] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:34.746760 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 7490
I20260928 21:12:34.747203 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:34.746803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44579 (local address 127.0.0.1:37314)
0928 21:12:34.747025 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:34.747093 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44579: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
W20260928 21:12:34.747736 7106 catalog_manager.cc:4584] TS f53c51d9d9104357bfa28537aa822e7f (127.0.98.1:44579): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fad3e1d7edfa444d96ef1a219c2950a8: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44579: connect: Connection refused (error 111)
I20260928 21:12:34.748585 7881 ts_tablet_manager.cc:1918] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:34.748677 7881 log.cc:1196] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/fad3e1d7edfa444d96ef1a219c2950a8
I20260928 21:12:34.750489 7881 ts_tablet_manager.cc:1939] T fad3e1d7edfa444d96ef1a219c2950a8 P 193c0b53f93243fbb0e499c350b63654: Deleting consensus metadata
I20260928 21:12:34.751439 7104 catalog_manager.cc:4841] TS 193c0b53f93243fbb0e499c350b63654 (127.0.98.3:38533): tablet fad3e1d7edfa444d96ef1a219c2950a8 (table test-table [id=18e185ca6bfa4580aff284fe37d34b9b]) successfully deleted
W20260928 21:12:34.751829 7104 catalog_manager.cc:4808] TS 193c0b53f93243fbb0e499c350b63654 (127.0.98.3:38533): delete failed for tablet fad3e1d7edfa444d96ef1a219c2950a8 because the tablet was not found. No further retry: Not found: Tablet not found: fad3e1d7edfa444d96ef1a219c2950a8
I20260928 21:12:34.753711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 7626
I20260928 21:12:34.760779 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 6983
I20260928 21:12:34.767326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 7091
2026-09-28T21:12:34Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7429 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:34 dist-test-slave-v7xz krb5kdc[7886](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:34 dist-test-slave-v7xz krb5kdc[7886](info): set up 2 sockets
Sep 28 21:12:34 dist-test-slave-v7xz krb5kdc[7886](info): commencing operation
krb5kdc: starting...
W20260928 21:12:35.837435 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:35 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:35Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260928 21:12:35.961948 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35255
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:36.071560 7899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:36.071868 7899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:36.071967 7899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:36.076081 7899 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:36.076169 7899 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260928 21:12:36.076203 7899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:36.076247 7899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:36.076287 7899 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:36.076324 7899 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:36.081610 7899 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35255
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:36.082733 7899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:36.083773 7899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:36.092748 7899 server_base.cc:1034] running on GCE node
W20260928 21:12:36.092695 7905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:36.092700 7904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:36.092674 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:36.093513 7899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:36.094673 7899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:36.095870 7899 hybrid_clock.cc:648] HybridClock initialized: now 1790629956095836 us; error 48 us; skew 500 ppm
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:36.098686 7899 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260928 21:12:36.099581 7899 webserver.cc:460] Webserver started at http://127.0.98.62:38823/ using document root <none> and password file <none>
I20260928 21:12:36.100135 7899 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:36.100224 7899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:36.100452 7899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:36.102515 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "37024f3e813b4e14b4d8e41d52243a98"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "f3c34c0f49162a8e6de47134656f7e80"
server_key_iv: "a87d6dd14273f8231b279b4b4fc2fc46"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.103070 7899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "37024f3e813b4e14b4d8e41d52243a98"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "f3c34c0f49162a8e6de47134656f7e80"
server_key_iv: "a87d6dd14273f8231b279b4b4fc2fc46"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.106705 7899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 21:12:36.109163 7914 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.110384 7899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:12:36.110558 7899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "37024f3e813b4e14b4d8e41d52243a98"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "f3c34c0f49162a8e6de47134656f7e80"
server_key_iv: "a87d6dd14273f8231b279b4b4fc2fc46"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.110687 7899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:36.121657 7899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:36.125319 7899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:36.125537 7899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:36.133423 7899 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35255
I20260928 21:12:36.133445 7976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35255 every 8 connection(s)
I20260928 21:12:36.134439 7899 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:36.137300 7977 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:36.138115 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 7899
I20260928 21:12:36.138258 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:36.138553 392 external_mini_cluster.cc:1428] Setting key d9e96625633c00a447ce5b1e4f4554aa
I20260928 21:12:36.144358 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Bootstrap starting.
I20260928 21:12:36.147269 7977 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:36.148043 7977 log.cc:824] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:36.150553 7977 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: No bootstrap required, opened a new log
I20260928 21:12:36.154691 7977 raft_consensus.cc:348] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37024f3e813b4e14b4d8e41d52243a98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35255 } }
I20260928 21:12:36.154982 7977 raft_consensus.cc:374] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:36.155038 7977 raft_consensus.cc:729] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37024f3e813b4e14b4d8e41d52243a98, State: Initialized, Role: FOLLOWER
I20260928 21:12:36.155405 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.140014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.0.98.62:35255)
0928 21:12:36.140467 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.140490 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.140538 (+ 48us) server_negotiation.cc:407] Connection header received
0928 21:12:36.141407 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.141434 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.141886 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.142285 (+ 399us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.143063 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.144573 (+ 1510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.145653 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.146384 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.146559 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.149292 (+ 2733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.149298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.149314 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.149343 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.152081 (+ 2738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.152555 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.152561 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.152568 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.152736 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.153081 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.153084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.153086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.153621 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.153837 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.154052 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.154359 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":116,"threads_started":1}
I20260928 21:12:36.155562 7977 consensus_queue.cc:260] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37024f3e813b4e14b4d8e41d52243a98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35255 } }
I20260928 21:12:36.155719 7977 raft_consensus.cc:388] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:36.155784 7977 raft_consensus.cc:482] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:36.155885 7977 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:36.157042 7977 raft_consensus.cc:504] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37024f3e813b4e14b4d8e41d52243a98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35255 } }
I20260928 21:12:36.157397 7977 leader_election.cc:302] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 37024f3e813b4e14b4d8e41d52243a98; no voters:
I20260928 21:12:36.157737 7977 leader_election.cc:288] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:36.157820 7982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:36.158032 7982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [term 1 LEADER]: Becoming Leader. State: Replica: 37024f3e813b4e14b4d8e41d52243a98, State: Running, Role: LEADER
I20260928 21:12:36.158384 7982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37024f3e813b4e14b4d8e41d52243a98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35255 } }
I20260928 21:12:36.159020 7977 sys_catalog.cc:564] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:36.160094 7983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "37024f3e813b4e14b4d8e41d52243a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37024f3e813b4e14b4d8e41d52243a98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35255 } } }
I20260928 21:12:36.160238 7983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:36.160159 7984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 37024f3e813b4e14b4d8e41d52243a98. Latest consensus state: current_term: 1 leader_uuid: "37024f3e813b4e14b4d8e41d52243a98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "37024f3e813b4e14b4d8e41d52243a98" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35255 } } }
I20260928 21:12:36.160368 7984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98 [sys.catalog]: This master's current role is: LEADER
W20260928 21:12:36.163940 7997 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:12:36.164033 7997 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:12:36.164104 7991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:36.164769 7991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:36.169921 7991 catalog_manager.cc:1347] Generated new cluster ID: 56f14190dac143818c5800b29cc20e04
I20260928 21:12:36.169991 7991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:36.175411 7991 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:36.176371 7991 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:36.181931 7991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Generated new TSK 0
I20260928 21:12:36.182660 7991 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260928 21:12:36.216717 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35255
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:36.337553 8003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:36.338021 8003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:36.338148 8003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:36.343277 8003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:36.343372 8003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:36.343447 8003 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:36.343537 8003 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:36.349108 8003 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:36.350308 8003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:36.351385 8003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:36.361465 8011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:36.361629 8003 server_base.cc:1034] running on GCE node
W20260928 21:12:36.361459 8008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:36.361466 8009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:36.362468 8003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:36.363139 8003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:36.364315 8003 hybrid_clock.cc:648] HybridClock initialized: now 1790629956364296 us; error 29 us; skew 500 ppm
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:36.367457 8003 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 21:12:36.368345 8003 webserver.cc:460] Webserver started at http://127.0.98.1:35035/ using document root <none> and password file <none>
I20260928 21:12:36.368969 8003 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:36.369062 8003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:36.369278 8003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:36.371315 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "09ea2943db314dbf8ed9bed70b51693e"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "7e462238fc11607e2661f0fc7092796e"
server_key_iv: "c75260988649fafa3d5718b493dd65c1"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.371869 8003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "09ea2943db314dbf8ed9bed70b51693e"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "7e462238fc11607e2661f0fc7092796e"
server_key_iv: "c75260988649fafa3d5718b493dd65c1"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.375694 8003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:12:36.378348 8018 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.379627 8003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:36.379765 8003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "09ea2943db314dbf8ed9bed70b51693e"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "7e462238fc11607e2661f0fc7092796e"
server_key_iv: "c75260988649fafa3d5718b493dd65c1"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.379879 8003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:36.395867 8003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:36.399386 8003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:36.399601 8003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:36.401160 8003 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:36.401242 8003 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.401314 8003 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:36.401356 8003 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:36.412379 8003 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34273
I20260928 21:12:36.412408 8133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34273 every 8 connection(s)
I20260928 21:12:36.413372 8003 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:36.414136 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8003
I20260928 21:12:36.414266 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:36.414613 392 external_mini_cluster.cc:1428] Setting key 546c0812d63b4a540c4bdad65ab85344
I20260928 21:12:36.416126 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.402097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38893 (local address 127.0.98.62:35255)
0928 21:12:36.402230 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.402234 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.403360 (+ 1126us) server_negotiation.cc:407] Connection header received
0928 21:12:36.404544 (+ 1184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.404548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.404620 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.404737 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.405977 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.406510 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.407957 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.408393 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.408538 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.411588 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.411591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.411593 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.411618 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.413285 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.414289 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.414296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.414298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.414416 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.414794 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.414796 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.414797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.415196 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.415302 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.415829 (+ 527us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.415970 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:12:36.417341 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.402601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.98.1:38893)
0928 21:12:36.403136 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.403178 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.404243 (+ 1065us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.404996 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.405004 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.405329 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:36.405813 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.405826 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.406636 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.406639 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.407835 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.407849 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.408579 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.408586 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.408814 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.409516 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:36.409531 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:36.411417 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:36.413402 (+ 1985us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.413411 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.413427 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.413789 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.414504 (+ 715us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.414508 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.414510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.414696 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.415299 (+ 603us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:36.415305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:36.415593 (+ 288us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.415786 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.416172 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":125,"threads_started":1}
I20260928 21:12:36.418782 8134 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35255
I20260928 21:12:36.418932 8134 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:36.419493 8134 heartbeater.cc:511] Master 127.0.98.62:35255 requested a full tablet report, sending...
I20260928 21:12:36.421099 7930 ts_manager.cc:194] Registered new tserver with Master: 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273)
I20260928 21:12:36.422214 7930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38893
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260928 21:12:36.431493 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.420974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42211 (local address 127.0.98.62:35255)
0928 21:12:36.421075 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.421079 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.424432 (+ 3353us) server_negotiation.cc:407] Connection header received
0928 21:12:36.424510 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.424512 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.424551 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.424621 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.425082 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.425593 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.426380 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.426728 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.426835 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.428034 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.428041 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.428048 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.428083 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.429686 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.430210 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.430213 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.430214 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.430287 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.430624 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.430627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.430630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.430823 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.430906 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.431280 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.431376 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:12:36.431625 8141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.420946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.98.1:42211)
0928 21:12:36.424340 (+ 3394us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.424353 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.424440 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.424662 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.424664 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.424820 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:36.424970 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.424976 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.425699 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.425701 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.426260 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.426267 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.426881 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.426888 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.426986 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.427569 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:36.427580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:36.427890 (+ 310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:36.429802 (+ 1912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.429809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.429814 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.430089 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.430383 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.430389 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.430394 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.430501 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.430927 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:36.430930 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:36.431041 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.431168 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.431339 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3315,"thread_start_us":107,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260928 21:12:36.441970 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35255
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:36.553993 8143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:36.554267 8143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:36.554314 8143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:36.558532 8143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:36.558593 8143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:36.558662 8143 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:36.558733 8143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:36.564175 8143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:36.565239 8143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:36.566294 8143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:36.576218 8148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:36.576303 8143 server_base.cc:1034] running on GCE node
W20260928 21:12:36.576237 8151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:36.576448 8149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:36.577247 8143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:36.578049 8143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:36.579231 8143 hybrid_clock.cc:648] HybridClock initialized: now 1790629956579198 us; error 49 us; skew 500 ppm
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:36.582480 8143 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 21:12:36.583328 8143 webserver.cc:460] Webserver started at http://127.0.98.2:44407/ using document root <none> and password file <none>
I20260928 21:12:36.583992 8143 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:36.584081 8143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:36.584295 8143 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:36.586367 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "703245b46b824f9a82bfd02dc02bbec9"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "e121ab09be71b34337b43679b9d0e2c3"
server_key_iv: "cd8eba271f403e1dbb0112cf456f8980"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.586921 8143 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "703245b46b824f9a82bfd02dc02bbec9"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "e121ab09be71b34337b43679b9d0e2c3"
server_key_iv: "cd8eba271f403e1dbb0112cf456f8980"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.590698 8143 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 21:12:36.593143 8158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.594302 8143 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:12:36.594431 8143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "703245b46b824f9a82bfd02dc02bbec9"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "e121ab09be71b34337b43679b9d0e2c3"
server_key_iv: "cd8eba271f403e1dbb0112cf456f8980"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.594538 8143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:36.622557 8143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:36.626184 8143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:36.626394 8143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:36.627933 8143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:36.628015 8143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.628082 8143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:36.628139 8143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:36.639454 8143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46079
I20260928 21:12:36.639855 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46079 every 8 connection(s)
I20260928 21:12:36.640475 8143 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:36.642513 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.628735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36937 (local address 127.0.98.62:35255)
0928 21:12:36.628895 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.628899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.630048 (+ 1149us) server_negotiation.cc:407] Connection header received
0928 21:12:36.631289 (+ 1241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.631293 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.631361 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.631483 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.633280 (+ 1797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.633970 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.635298 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.635540 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.635656 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.638445 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.638448 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.638450 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.638499 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.640389 (+ 1890us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.641012 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.641018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.641019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.641121 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.641553 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.641556 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.641557 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.641747 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.641830 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.642294 (+ 464us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.642392 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:12:36.643343 8167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.629354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.98.2:36937)
0928 21:12:36.629865 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.629912 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.631018 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.631736 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.631751 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.632262 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:36.633086 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.633110 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.634109 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.634113 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.635159 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.635169 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.635713 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.635717 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.635902 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.636598 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:36.636614 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:36.638260 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:36.640529 (+ 2269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.640535 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.640550 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.640812 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.641227 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.641230 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.641235 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.641450 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.641844 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:36.641849 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:36.642078 (+ 229us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.642225 (+ 147us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.642472 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":127,"threads_started":1}
I20260928 21:12:36.644719 8274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35255
I20260928 21:12:36.644901 8274 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:36.645406 8274 heartbeater.cc:511] Master 127.0.98.62:35255 requested a full tablet report, sending...
I20260928 21:12:36.646431 7931 ts_manager.cc:194] Registered new tserver with Master: 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079)
I20260928 21:12:36.647013 7931 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36937
I20260928 21:12:36.648897 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8143
I20260928 21:12:36.649046 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:36.649379 392 external_mini_cluster.cc:1428] Setting key cb0b8123945b99691d9e1c5393fac8e9
I20260928 21:12:36.654778 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.647008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40895 (local address 127.0.98.62:35255)
0928 21:12:36.647154 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.647160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.647438 (+ 278us) server_negotiation.cc:407] Connection header received
0928 21:12:36.647546 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.647548 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.647611 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.647700 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.648100 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.648809 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.649945 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.650309 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.650405 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.651580 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.651586 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.651593 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.651625 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.653125 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.653544 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.653549 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.653551 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.653630 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.653905 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.653910 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.653912 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.654120 (+ 208us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.654235 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.654500 (+ 265us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.654601 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":70}
I20260928 21:12:36.654789 8280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.646924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.98.2:40895)
0928 21:12:36.647335 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.647351 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.647450 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.647722 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.647724 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.647891 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:36.647990 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.647996 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.648952 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.648956 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.649826 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.649835 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.650421 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.650424 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.650532 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.651105 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:36.651116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:36.651443 (+ 327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:36.653233 (+ 1790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.653238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.653241 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.653440 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.653729 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.653732 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.653734 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.653818 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.654224 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:36.654250 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:36.654342 (+ 92us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.654446 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.654599 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":127,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 21:12:36.676724 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35255
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:36.788708 8283 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:36.789073 8283 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:36.789162 8283 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:36.793380 8283 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:36.793464 8283 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:36.793538 8283 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260928 21:12:36.793617 8283 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:36.798955 8283 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:35255
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8283
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:36.800010 8283 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:36.801045 8283 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:36.810933 8288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:36.810979 8291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:36.810935 8289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:36.811162 8283 server_base.cc:1034] running on GCE node
I20260928 21:12:36.811630 8283 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:36.812376 8283 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:36.813526 8283 hybrid_clock.cc:648] HybridClock initialized: now 1790629956813492 us; error 53 us; skew 500 ppm
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:36.816566 8283 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 21:12:36.817553 8283 webserver.cc:460] Webserver started at http://127.0.98.3:36413/ using document root <none> and password file <none>
I20260928 21:12:36.818089 8283 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:36.818161 8283 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:36.818416 8283 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:36.820446 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2db6797d0c34a7a8f257a14b2e70f26"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "99eafa3e62ec1494f83446ad9f0d671a"
server_key_iv: "3ba7f74c6c683bd98968fa527065e63d"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.821020 8283 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2db6797d0c34a7a8f257a14b2e70f26"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "99eafa3e62ec1494f83446ad9f0d671a"
server_key_iv: "3ba7f74c6c683bd98968fa527065e63d"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.824745 8283 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:36.827203 8298 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.828486 8283 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:36.828616 8283 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "f2db6797d0c34a7a8f257a14b2e70f26"
format_stamp: "Formatted at 2026-09-28 21:12:36 on dist-test-slave-v7xz"
server_key: "99eafa3e62ec1494f83446ad9f0d671a"
server_key_iv: "3ba7f74c6c683bd98968fa527065e63d"
server_key_version: "encryptionkey@0"
I20260928 21:12:36.828727 8283 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:36.847424 8283 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:36.850471 8283 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:36.850684 8283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:36.852212 8283 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:36.852288 8283 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:36.852362 8283 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:36.852411 8283 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:36.863440 8283 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43577
I20260928 21:12:36.863516 8413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43577 every 8 connection(s)
I20260928 21:12:36.864315 8283 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:36.865869 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.853188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44467 (local address 127.0.98.62:35255)
0928 21:12:36.853317 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.853321 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.854346 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 21:12:36.855523 (+ 1177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.855527 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.855586 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.855674 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.857005 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.857523 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.858963 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.859205 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.859323 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.862166 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.862169 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.862172 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.862207 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.863859 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.864369 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.864373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.864375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.864467 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.864878 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.864881 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.864883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.865109 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.865217 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.865658 (+ 441us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.865754 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:12:36.866675 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.853703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.98.3:44467)
0928 21:12:36.854194 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.854235 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.855228 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.855860 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.855873 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.856265 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:36.856804 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.856819 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.857648 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.857652 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.858825 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.858834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.859370 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.859374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.859556 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.860297 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:36.860313 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:36.861992 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:36.863978 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.863984 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.864000 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.864263 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.864568 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.864571 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.864573 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.864740 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.865227 (+ 487us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:36.865233 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:36.865451 (+ 218us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.865611 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.865856 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":125,"threads_started":1}
I20260928 21:12:36.867799 8414 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35255
I20260928 21:12:36.867949 8414 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:36.868536 8414 heartbeater.cc:511] Master 127.0.98.62:35255 requested a full tablet report, sending...
I20260928 21:12:36.869676 7930 ts_manager.cc:194] Registered new tserver with Master: f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577)
I20260928 21:12:36.870417 7930 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44467
I20260928 21:12:36.873618 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8283
I20260928 21:12:36.873775 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:36.874114 392 external_mini_cluster.cc:1428] Setting key b3c0d01448c63ebed21e6c87b5274d30
I20260928 21:12:36.876439 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:12:36.877957 8420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.870237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.98.3:47157)
0928 21:12:36.870519 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.870537 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.870663 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.871187 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.871190 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.871409 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:36.871511 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.871517 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.872308 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.872311 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.872999 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.873008 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.873528 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.873531 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.873601 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.874161 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:36.874172 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:36.874464 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:36.876280 (+ 1816us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.876287 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.876289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.876546 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.876872 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:36.876875 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:36.876878 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:36.876981 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:36.877384 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:36.877388 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:36.877499 (+ 111us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.877618 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.877762 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":87,"threads_started":1}
I20260928 21:12:36.877975 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.870340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47157 (local address 127.0.98.62:35255)
0928 21:12:36.870499 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.870504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.870618 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:12:36.871025 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.871028 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.871069 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.871171 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.871632 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.872176 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.873122 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.873415 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.873533 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.874600 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.874606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.874611 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.874640 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.876155 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.876637 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.876642 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.876644 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.876733 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.877094 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.877098 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.877100 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.877277 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.877396 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.877675 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.877815 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:12:36.901916 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:36.895717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.0.98.62:35255)
0928 21:12:36.895848 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.895851 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.895862 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:12:36.895898 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.895900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.895935 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.896043 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.896738 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.897325 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.898223 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.898440 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.898553 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.901768 (+ 3215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35646: BlockingRecv error: recv got EOF from 127.0.0.1:35646 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 28 21:12:36 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:36.911621 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.903341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.0.98.62:35255)
0928 21:12:36.903475 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.903479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.903598 (+ 119us) server_negotiation.cc:407] Connection header received
0928 21:12:36.903746 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.903749 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.903798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.903881 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:36.904451 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.905042 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.905846 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.906072 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.906183 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.908451 (+ 2268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:36.908455 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:36.908457 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:36.908480 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:36.909979 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.910407 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.910411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.910413 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.910495 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:36.910776 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:36.910780 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:36.910782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:36.910960 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:36.911103 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.911340 (+ 237us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.911451 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:12:36.914513 7930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:36.917604 7930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:36.932636 8436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.926383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46079 (local address 127.0.0.1:60762)
0928 21:12:36.927652 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.927667 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.927807 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.928496 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.928498 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.928524 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:36.928771 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.928778 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.929800 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.929806 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.930979 (+ 1173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.930987 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.931977 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.931981 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.932086 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.932122 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.932275 (+ 153us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.932496 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":91,"spinlock_wait_cycles":896,"thread_start_us":77,"threads_started":1}
I20260928 21:12:36.932636 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.926588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60762 (local address 127.0.98.2:46079)
0928 21:12:36.927155 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.927169 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.927741 (+ 572us) server_negotiation.cc:407] Connection header received
0928 21:12:36.927963 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.927970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.928199 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.928321 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:36.928915 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.929687 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.931101 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.931865 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.931980 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.932165 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.932308 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.932405 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":112,"threads_started":1}
I20260928 21:12:36.933485 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.927221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43577 (local address 127.0.0.1:60204)
0928 21:12:36.927958 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.927970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.928058 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.928518 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.928520 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.928533 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:36.928782 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.928787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.930265 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.930268 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.931546 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.931554 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.933058 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.933061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.933147 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.933169 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.933248 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.933315 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"mutex_wait_us":64,"spinlock_wait_cycles":2816,"thread_start_us":92,"threads_started":1}
I20260928 21:12:36.933939 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.927392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60204 (local address 127.0.98.3:43577)
0928 21:12:36.927686 (+ 294us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.927702 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.928157 (+ 455us) server_negotiation.cc:407] Connection header received
0928 21:12:36.928231 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.928236 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.928392 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.928526 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:36.928944 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.930131 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.931678 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.932705 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.932801 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.932995 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.933671 (+ 676us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.933779 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":89,"threads_started":1}
I20260928 21:12:36.936491 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 53fea4387de4411f96e34e5c2278010b (DEFAULT_TABLE table=test-table [id=9dca132651ef4496a6dda5adc90cef0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:36.936795 8346 tablet_service.cc:1467] Processing CreateTablet for tablet 53fea4387de4411f96e34e5c2278010b (DEFAULT_TABLE table=test-table [id=9dca132651ef4496a6dda5adc90cef0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:36.937307 8437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.926560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34273 (local address 127.0.0.1:54270)
0928 21:12:36.927499 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:36.927579 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:36.927777 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:36.928457 (+ 680us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:36.928461 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:36.928508 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:36.928772 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.928781 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.931422 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.931425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.932570 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:36.932581 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.933946 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.933952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:36.934063 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.934085 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:12:36.937016 (+ 2931us) client_negotiation.cc:240] Negotiation successful
0928 21:12:36.937152 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":863,"mutex_wait_us":90,"spinlock_wait_cycles":20608,"thread_start_us":129,"threads_started":1}
I20260928 21:12:36.937712 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:36.926736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54270 (local address 127.0.98.1:34273)
0928 21:12:36.927345 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:36.927351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:36.927668 (+ 317us) server_negotiation.cc:407] Connection header received
0928 21:12:36.927992 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:36.927998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:36.928198 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:36.928297 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:36.928937 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.930096 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.932719 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:36.933645 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:36.933722 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:36.933846 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:36.937049 (+ 3203us) server_negotiation.cc:299] Negotiation successful
0928 21:12:36.937252 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":71,"threads_started":1}
I20260928 21:12:36.937752 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fea4387de4411f96e34e5c2278010b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:36.937743 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fea4387de4411f96e34e5c2278010b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:36.940330 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 53fea4387de4411f96e34e5c2278010b (DEFAULT_TABLE table=test-table [id=9dca132651ef4496a6dda5adc90cef0b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:36.941359 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53fea4387de4411f96e34e5c2278010b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:36.946303 8443 tablet_bootstrap.cc:492] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Bootstrap starting.
I20260928 21:12:36.946705 8442 tablet_bootstrap.cc:492] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Bootstrap starting.
I20260928 21:12:36.948603 8444 tablet_bootstrap.cc:492] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Bootstrap starting.
I20260928 21:12:36.950093 8443 tablet_bootstrap.cc:654] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:36.950713 8442 tablet_bootstrap.cc:654] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:36.950850 8444 tablet_bootstrap.cc:654] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:36.950985 8443 log.cc:824] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:36.951493 8442 log.cc:824] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:36.951648 8444 log.cc:824] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:36.953229 8443 tablet_bootstrap.cc:492] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: No bootstrap required, opened a new log
I20260928 21:12:36.953356 8444 tablet_bootstrap.cc:492] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: No bootstrap required, opened a new log
I20260928 21:12:36.953433 8442 tablet_bootstrap.cc:492] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: No bootstrap required, opened a new log
I20260928 21:12:36.953440 8443 ts_tablet_manager.cc:1397] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:12:36.953583 8444 ts_tablet_manager.cc:1397] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:12:36.953598 8442 ts_tablet_manager.cc:1397] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:12:36.957257 8444 raft_consensus.cc:348] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:36.957507 8444 raft_consensus.cc:374] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:36.957559 8444 raft_consensus.cc:729] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea2943db314dbf8ed9bed70b51693e, State: Initialized, Role: FOLLOWER
I20260928 21:12:36.957386 8442 raft_consensus.cc:348] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:36.957630 8442 raft_consensus.cc:374] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:36.957706 8442 raft_consensus.cc:729] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Initialized, Role: FOLLOWER
I20260928 21:12:36.957641 8443 raft_consensus.cc:348] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:36.957861 8443 raft_consensus.cc:374] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:36.957919 8443 raft_consensus.cc:729] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Initialized, Role: FOLLOWER
I20260928 21:12:36.958122 8444 consensus_queue.cc:260] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:36.958184 8442 consensus_queue.cc:260] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:36.958257 8443 consensus_queue.cc:260] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:36.958829 8414 heartbeater.cc:503] Master 127.0.98.62:35255 was elected leader, sending a full tablet report...
I20260928 21:12:36.959142 8443 ts_tablet_manager.cc:1428] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:12:36.959142 8442 ts_tablet_manager.cc:1428] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:12:36.959172 8444 ts_tablet_manager.cc:1428] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:12:36.959463 8134 heartbeater.cc:503] Master 127.0.98.62:35255 was elected leader, sending a full tablet report...
I20260928 21:12:36.959463 8274 heartbeater.cc:503] Master 127.0.98.62:35255 was elected leader, sending a full tablet report...
I20260928 21:12:37.011425 8449 raft_consensus.cc:482] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:37.011684 8449 raft_consensus.cc:504] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.012903 8449 leader_election.cc:288] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079), 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273)
I20260928 21:12:37.017035 8307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.013126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46079 (local address 127.0.98.3:42807)
0928 21:12:37.013333 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.013352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.013462 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.013761 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.013764 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.013787 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:37.013917 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.013927 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.014797 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.014801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.015742 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.015751 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.016592 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.016596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.016693 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.016719 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.016797 (+ 78us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.016899 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88}
I20260928 21:12:37.017030 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.013224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42807 (local address 127.0.98.2:46079)
0928 21:12:37.013347 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.013352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.013429 (+ 77us) server_negotiation.cc:407] Connection header received
0928 21:12:37.013585 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.013588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.013653 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.013736 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:37.014036 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.014695 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.015867 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.016484 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.016560 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.016612 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.016817 (+ 205us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.016925 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:12:37.017673 8451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.013210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34273 (local address 127.0.98.3:54007)
0928 21:12:37.013605 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.013617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.013728 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.013962 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.013964 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.013999 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:37.014139 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.014146 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.014940 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.014942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.016337 (+ 1395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.016347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.017329 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.017332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.017428 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.017456 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.017516 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.017565 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":29,"thread_start_us":132,"threads_started":1}
I20260928 21:12:37.017729 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.013313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54007 (local address 127.0.98.1:34273)
0928 21:12:37.013430 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.013435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.013729 (+ 294us) server_negotiation.cc:407] Connection header received
0928 21:12:37.013810 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.013813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.013861 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.013945 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:37.014258 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.014834 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.016462 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.017095 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.017200 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.017268 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.017538 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.017611 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:12:37.017760 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fea4387de4411f96e34e5c2278010b" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703245b46b824f9a82bfd02dc02bbec9" is_pre_election: true
I20260928 21:12:37.018131 8228 raft_consensus.cc:2454] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 0.
I20260928 21:12:37.018458 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fea4387de4411f96e34e5c2278010b" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e" is_pre_election: true
I20260928 21:12:37.018682 8299 leader_election.cc:302] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.018754 8088 raft_consensus.cc:2454] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 0.
I20260928 21:12:37.018960 8449 raft_consensus.cc:2781] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:37.019068 8449 raft_consensus.cc:482] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:37.019129 8449 raft_consensus.cc:3037] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.020215 8449 raft_consensus.cc:504] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.020604 8449 leader_election.cc:288] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 election: Requested vote from peers 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079), 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273)
I20260928 21:12:37.020987 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fea4387de4411f96e34e5c2278010b" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703245b46b824f9a82bfd02dc02bbec9"
I20260928 21:12:37.021056 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53fea4387de4411f96e34e5c2278010b" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e"
I20260928 21:12:37.021145 8228 raft_consensus.cc:3037] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.021202 8088 raft_consensus.cc:3037] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.022179 8228 raft_consensus.cc:2454] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 1.
I20260928 21:12:37.022244 8088 raft_consensus.cc:2454] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 1.
I20260928 21:12:37.022483 8299 leader_election.cc:302] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.022661 8449 raft_consensus.cc:2781] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:37.022881 8449 raft_consensus.cc:686] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 LEADER]: Becoming Leader. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Running, Role: LEADER
I20260928 21:12:37.023269 8449 consensus_queue.cc:237] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.026548 7930 catalog_manager.cc:5495] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } health_report { overall_health: HEALTHY } } }
I20260928 21:12:37.055516 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.052337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60208 (local address 127.0.98.3:43577)
0928 21:12:37.052460 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.052464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.052477 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:37.052552 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.052554 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.052607 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.052718 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:37.052944 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.053430 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.054262 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.054461 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.054535 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.054761 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.055197 (+ 436us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.055293 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.055351 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.055403 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:37.061734 8088 raft_consensus.cc:1261] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Refusing update from remote peer f2db6797d0c34a7a8f257a14b2e70f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.062223 8228 raft_consensus.cc:1261] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Refusing update from remote peer f2db6797d0c34a7a8f257a14b2e70f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.062546 8452 consensus_queue.cc:1035] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.062880 8449 consensus_queue.cc:1035] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.069855 8455 mvcc.cc:205] Tried to move back new op lower bound from 7334420304116420608 to 7334420303970332672. Current Snapshot: MvccSnapshot[applied={T|T < 7334420304116420608}]
I20260928 21:12:37.070492 8456 mvcc.cc:205] Tried to move back new op lower bound from 7334420304116420608 to 7334420303970332672. Current Snapshot: MvccSnapshot[applied={T|T < 7334420304116420608}]
I20260928 21:12:37.078619 7930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.078773 7930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.081270 7930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Sending DeleteTablet for 3 replicas of tablet 53fea4387de4411f96e34e5c2278010b
I20260928 21:12:37.082093 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fea4387de4411f96e34e5c2278010b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60762
I20260928 21:12:37.082115 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fea4387de4411f96e34e5c2278010b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260928 21:12:37.082512 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet 53fea4387de4411f96e34e5c2278010b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60204
I20260928 21:12:37.083063 8466 tablet_replica.cc:330] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: stopping tablet replica
I20260928 21:12:37.083112 8465 tablet_replica.cc:330] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: stopping tablet replica
I20260928 21:12:37.083362 7930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:12:37.083448 8465 raft_consensus.cc:2229] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.083483 8466 raft_consensus.cc:2229] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:37.083781 8465 raft_consensus.cc:2258] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:12:37.083793 7930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:37.084193 8464 tablet_replica.cc:330] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: stopping tablet replica
I20260928 21:12:37.084460 8464 raft_consensus.cc:2229] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.084733 8464 raft_consensus.cc:2258] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.085696 8465 ts_tablet_manager.cc:1905] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.087599 8464 ts_tablet_manager.cc:1905] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.088652 8465 ts_tablet_manager.cc:1918] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:37.088817 8465 log.cc:1196] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/53fea4387de4411f96e34e5c2278010b
I20260928 21:12:37.089260 8465 ts_tablet_manager.cc:1939] T 53fea4387de4411f96e34e5c2278010b P 703245b46b824f9a82bfd02dc02bbec9: Deleting consensus metadata
I20260928 21:12:37.089370 8346 tablet_service.cc:1467] Processing CreateTablet for tablet b5ebf652c747447eae6593ea4fd4d3f1 (DEFAULT_TABLE table=test-table [id=44d02bda7444446890039b137f84d29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.089401 8067 tablet_service.cc:1467] Processing CreateTablet for tablet b5ebf652c747447eae6593ea4fd4d3f1 (DEFAULT_TABLE table=test-table [id=44d02bda7444446890039b137f84d29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.089634 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ebf652c747447eae6593ea4fd4d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.089663 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ebf652c747447eae6593ea4fd4d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.090171 8208 tablet_service.cc:1467] Processing CreateTablet for tablet b5ebf652c747447eae6593ea4fd4d3f1 (DEFAULT_TABLE table=test-table [id=44d02bda7444446890039b137f84d29b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.090413 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b5ebf652c747447eae6593ea4fd4d3f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.090960 7915 catalog_manager.cc:4841] TS 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079): tablet 53fea4387de4411f96e34e5c2278010b (table test-table [id=9dca132651ef4496a6dda5adc90cef0b]) successfully deleted
I20260928 21:12:37.092345 8443 tablet_bootstrap.cc:492] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Bootstrap starting.
I20260928 21:12:37.092351 8442 tablet_bootstrap.cc:492] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Bootstrap starting.
I20260928 21:12:37.093318 8442 tablet_bootstrap.cc:654] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.093771 8443 tablet_bootstrap.cc:654] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.095008 8443 tablet_bootstrap.cc:492] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: No bootstrap required, opened a new log
I20260928 21:12:37.095093 8443 ts_tablet_manager.cc:1397] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:37.095127 8464 ts_tablet_manager.cc:1918] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:37.095221 8464 log.cc:1196] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/53fea4387de4411f96e34e5c2278010b
I20260928 21:12:37.095511 8464 ts_tablet_manager.cc:1939] T 53fea4387de4411f96e34e5c2278010b P 09ea2943db314dbf8ed9bed70b51693e: Deleting consensus metadata
I20260928 21:12:37.095563 8443 raft_consensus.cc:348] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.095721 8443 raft_consensus.cc:374] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.095757 8443 raft_consensus.cc:729] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.095883 8443 consensus_queue.cc:260] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.095958 8466 raft_consensus.cc:2258] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.096135 8443 ts_tablet_manager.cc:1428] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.096318 8442 tablet_bootstrap.cc:492] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: No bootstrap required, opened a new log
I20260928 21:12:37.096338 8444 tablet_bootstrap.cc:492] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Bootstrap starting.
I20260928 21:12:37.096391 8442 ts_tablet_manager.cc:1397] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:12:37.096848 8442 raft_consensus.cc:348] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.096951 8442 raft_consensus.cc:374] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.096982 8442 raft_consensus.cc:729] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.097093 8442 consensus_queue.cc:260] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.097273 8444 tablet_bootstrap.cc:654] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.097313 8442 ts_tablet_manager.cc:1428] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.099753 7916 catalog_manager.cc:4841] TS 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273): tablet 53fea4387de4411f96e34e5c2278010b (table test-table [id=9dca132651ef4496a6dda5adc90cef0b]) successfully deleted
I20260928 21:12:37.100946 8466 ts_tablet_manager.cc:1905] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.103339 8466 ts_tablet_manager.cc:1918] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:37.103436 8466 log.cc:1196] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/53fea4387de4411f96e34e5c2278010b
I20260928 21:12:37.103725 8466 ts_tablet_manager.cc:1939] T 53fea4387de4411f96e34e5c2278010b P f2db6797d0c34a7a8f257a14b2e70f26: Deleting consensus metadata
I20260928 21:12:37.104709 8444 tablet_bootstrap.cc:492] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: No bootstrap required, opened a new log
I20260928 21:12:37.104802 8444 ts_tablet_manager.cc:1397] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260928 21:12:37.105581 8444 raft_consensus.cc:348] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.105721 8444 raft_consensus.cc:374] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.105754 8444 raft_consensus.cc:729] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea2943db314dbf8ed9bed70b51693e, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.106002 7918 catalog_manager.cc:4841] TS f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577): tablet 53fea4387de4411f96e34e5c2278010b (table test-table [id=9dca132651ef4496a6dda5adc90cef0b]) successfully deleted
I20260928 21:12:37.106055 8444 consensus_queue.cc:260] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.106412 8444 ts_tablet_manager.cc:1428] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20260928 21:12:37.115199 8415 tablet.cc:2369] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:37.142055 8275 tablet.cc:2369] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:12:37.164913 8135 tablet.cc:2369] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:12:37.237318 8452 raft_consensus.cc:482] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:37.237491 8452 raft_consensus.cc:504] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.237959 8452 leader_election.cc:288] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079)
I20260928 21:12:37.238476 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ebf652c747447eae6593ea4fd4d3f1" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703245b46b824f9a82bfd02dc02bbec9" is_pre_election: true
I20260928 21:12:37.238476 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ebf652c747447eae6593ea4fd4d3f1" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e" is_pre_election: true
I20260928 21:12:37.238636 8088 raft_consensus.cc:2454] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 0.
I20260928 21:12:37.238636 8228 raft_consensus.cc:2454] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 0.
I20260928 21:12:37.238946 8299 leader_election.cc:302] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.239172 8452 raft_consensus.cc:2781] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:37.239239 8452 raft_consensus.cc:482] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:37.239269 8452 raft_consensus.cc:3037] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.240219 8452 raft_consensus.cc:504] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.240536 8452 leader_election.cc:288] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 election: Requested vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079)
I20260928 21:12:37.240916 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ebf652c747447eae6593ea4fd4d3f1" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e"
I20260928 21:12:37.241024 8088 raft_consensus.cc:3037] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.240983 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b5ebf652c747447eae6593ea4fd4d3f1" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703245b46b824f9a82bfd02dc02bbec9"
I20260928 21:12:37.241132 8228 raft_consensus.cc:3037] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.241986 8088 raft_consensus.cc:2454] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 1.
I20260928 21:12:37.242115 8228 raft_consensus.cc:2454] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 1.
I20260928 21:12:37.242326 8300 leader_election.cc:302] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea2943db314dbf8ed9bed70b51693e, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.242556 8452 raft_consensus.cc:2781] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:37.242654 8452 raft_consensus.cc:686] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 LEADER]: Becoming Leader. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Running, Role: LEADER
I20260928 21:12:37.242801 8452 consensus_queue.cc:237] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.244428 7930 catalog_manager.cc:5495] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } health_report { overall_health: HEALTHY } } }
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629956, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:37.279592 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.270584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35255 (local address 127.0.0.1:35668)
0928 21:12:37.270920 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.270937 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.271059 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.271338 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.271341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.271774 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:37.271872 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.271878 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.272645 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.272648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.273376 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.273385 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.273811 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.273814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.273877 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.274574 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:37.274586 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:37.276095 (+ 1509us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:37.277932 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:37.277942 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:37.277950 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:37.278198 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:37.278483 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:37.278486 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:37.278488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:37.278566 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:37.278950 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:37.278959 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:37.279076 (+ 117us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.279195 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.279448 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":100,"thread_start_us":134,"threads_started":1}
I20260928 21:12:37.279590 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.270673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35668 (local address 127.0.98.62:35255)
0928 21:12:37.270807 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.270810 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.271071 (+ 261us) server_negotiation.cc:407] Connection header received
0928 21:12:37.271131 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.271134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.271176 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.271263 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:37.272001 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.272514 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.273495 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.273702 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.273784 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.276250 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:37.276253 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:37.276255 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:37.276275 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:37.277794 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:37.278303 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:37.278309 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:37.278311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:37.278386 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:37.278659 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:37.278663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:37.278667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:37.278847 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:37.278950 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.279238 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.279395 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":52}
I20260928 21:12:37.284101 7930 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35668:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260928 21:12:37.284868 7930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:37.289752 8067 tablet_service.cc:1467] Processing CreateTablet for tablet c042f7071b4247939bbea194f7a8a651 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8731551476214879a5671db56e5531c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:37.289752 8208 tablet_service.cc:1467] Processing CreateTablet for tablet c042f7071b4247939bbea194f7a8a651 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8731551476214879a5671db56e5531c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:37.289832 8346 tablet_service.cc:1467] Processing CreateTablet for tablet c042f7071b4247939bbea194f7a8a651 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=8731551476214879a5671db56e5531c0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260928 21:12:37.290205 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c042f7071b4247939bbea194f7a8a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.290217 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c042f7071b4247939bbea194f7a8a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.290222 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c042f7071b4247939bbea194f7a8a651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.292785 8444 tablet_bootstrap.cc:492] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: Bootstrap starting.
I20260928 21:12:37.292994 8443 tablet_bootstrap.cc:492] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: Bootstrap starting.
I20260928 21:12:37.293327 8442 tablet_bootstrap.cc:492] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: Bootstrap starting.
I20260928 21:12:37.293970 8444 tablet_bootstrap.cc:654] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.294129 8443 tablet_bootstrap.cc:654] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.294493 8442 tablet_bootstrap.cc:654] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.295127 8444 tablet_bootstrap.cc:492] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: No bootstrap required, opened a new log
I20260928 21:12:37.295225 8444 ts_tablet_manager.cc:1397] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:37.295516 8443 tablet_bootstrap.cc:492] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: No bootstrap required, opened a new log
I20260928 21:12:37.295601 8443 ts_tablet_manager.cc:1397] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:12:37.295728 8444 raft_consensus.cc:348] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.295849 8444 raft_consensus.cc:374] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.295884 8444 raft_consensus.cc:729] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea2943db314dbf8ed9bed70b51693e, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.296015 8444 consensus_queue.cc:260] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.296051 8443 raft_consensus.cc:348] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.296144 8443 raft_consensus.cc:374] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.296175 8443 raft_consensus.cc:729] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.296298 8443 consensus_queue.cc:260] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.296327 8444 ts_tablet_manager.cc:1428] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:12:37.296303 8448 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.296489 8448 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.296582 8443 ts_tablet_manager.cc:1428] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.296818 8450 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.296993 8450 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.297096 8442 tablet_bootstrap.cc:492] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: No bootstrap required, opened a new log
I20260928 21:12:37.297173 8442 ts_tablet_manager.cc:1397] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260928 21:12:37.297597 8442 raft_consensus.cc:348] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.297704 8442 raft_consensus.cc:374] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.297732 8442 raft_consensus.cc:729] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.297897 8442 consensus_queue.cc:260] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.298286 8452 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.298411 8452 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.298813 8442 ts_tablet_manager.cc:1428] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:37.464593 8450 raft_consensus.cc:482] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:37.464744 8450 raft_consensus.cc:504] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.465884 8450 leader_election.cc:288] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577)
I20260928 21:12:37.471128 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.466291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36915 (local address 127.0.98.3:43577)
0928 21:12:37.466416 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.466421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.466569 (+ 148us) server_negotiation.cc:407] Connection header received
0928 21:12:37.466676 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.466679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.466727 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.466814 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:37.467167 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.467927 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.469824 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.470434 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.470548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.470613 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.470926 (+ 313us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.470995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:12:37.471128 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.466214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46009 (local address 127.0.98.1:34273)
0928 21:12:37.466445 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.466449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.466522 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:12:37.466676 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.466679 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.466727 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.466814 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:12:37.467157 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.467951 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.469824 (+ 1873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.470434 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.470548 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.470611 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.470926 (+ 315us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.470995 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260928 21:12:37.471184 8480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.466167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43577 (local address 127.0.98.2:36915)
0928 21:12:37.466487 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.466500 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.466569 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.466835 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.466838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.466876 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:37.467047 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.467060 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.468087 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.468089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.469691 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.469703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.470595 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.470599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.470762 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.470807 (+ 45us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.470895 (+ 88us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.470999 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":180,"spinlock_wait_cycles":29824,"thread_start_us":63,"threads_started":1}
I20260928 21:12:37.471194 8481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.466136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34273 (local address 127.0.98.2:46009)
0928 21:12:37.466402 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.466418 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.466569 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.466835 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.466838 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.466878 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:12:37.467031 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.467039 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.468081 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.468085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.469691 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.469703 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.470595 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.470599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.470758 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.470814 (+ 56us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.470895 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.470996 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":173,"spinlock_wait_cycles":11776,"thread_start_us":94,"threads_started":1}
I20260928 21:12:37.471732 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c042f7071b4247939bbea194f7a8a651" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e" is_pre_election: true
I20260928 21:12:37.471772 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c042f7071b4247939bbea194f7a8a651" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" is_pre_election: true
I20260928 21:12:37.471868 8088 raft_consensus.cc:2454] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 0.
I20260928 21:12:37.472059 8368 raft_consensus.cc:2454] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 0.
I20260928 21:12:37.472339 8160 leader_election.cc:302] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea2943db314dbf8ed9bed70b51693e, 703245b46b824f9a82bfd02dc02bbec9; no voters:
I20260928 21:12:37.472700 8450 raft_consensus.cc:2781] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:37.472756 8450 raft_consensus.cc:482] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:37.472783 8450 raft_consensus.cc:3037] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.473842 8450 raft_consensus.cc:504] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.474252 8450 leader_election.cc:288] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 election: Requested vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577)
I20260928 21:12:37.474586 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c042f7071b4247939bbea194f7a8a651" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2db6797d0c34a7a8f257a14b2e70f26"
I20260928 21:12:37.474596 8088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c042f7071b4247939bbea194f7a8a651" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e"
I20260928 21:12:37.474699 8368 raft_consensus.cc:3037] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.474730 8088 raft_consensus.cc:3037] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.475616 8368 raft_consensus.cc:2454] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 1.
I20260928 21:12:37.475627 8088 raft_consensus.cc:2454] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 1.
I20260928 21:12:37.475908 8162 leader_election.cc:302] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.476104 8450 raft_consensus.cc:2781] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:37.476327 8450 raft_consensus.cc:686] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 LEADER]: Becoming Leader. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Running, Role: LEADER
I20260928 21:12:37.476572 8450 consensus_queue.cc:237] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.477816 8482 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: TxnStatusTablet state changed. Reason: New leader 703245b46b824f9a82bfd02dc02bbec9. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.477929 8482 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:37.478209 8484 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260928 21:12:37.478349 8484 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260928 21:12:37.479064 7931 catalog_manager.cc:5495] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:37.536276 8476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.532375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46079 (local address 127.0.0.1:60772)
0928 21:12:37.532527 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.532549 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.532672 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.533122 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.533125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.533135 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:37.533244 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.533249 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.534029 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.534032 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.534651 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.534659 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.535137 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.535141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.535232 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.535274 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.535966 (+ 692us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.535983 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.536054 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.536140 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260928 21:12:37.536298 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.532480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60772 (local address 127.0.98.2:46079)
0928 21:12:37.532721 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.532728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.532746 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:12:37.532919 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.532921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.532991 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.533109 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:37.533350 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.533905 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.534769 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.535022 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.535137 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.535433 (+ 296us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.535857 (+ 424us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.535997 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.536062 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.536114 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":67,"threads_started":1}
I20260928 21:12:37.539354 8368 raft_consensus.cc:1261] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Refusing update from remote peer 703245b46b824f9a82bfd02dc02bbec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.539740 8088 raft_consensus.cc:1261] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Refusing update from remote peer 703245b46b824f9a82bfd02dc02bbec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.540031 8482 consensus_queue.cc:1035] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.540300 8450 consensus_queue.cc:1035] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.541723 8448 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: TxnStatusTablet state changed. Reason: New leader 703245b46b824f9a82bfd02dc02bbec9. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.541841 8448 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.542199 8452 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: TxnStatusTablet state changed. Reason: New leader 703245b46b824f9a82bfd02dc02bbec9. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.542306 8452 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.543733 8450 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.543869 8450 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:37.543983 8457 mvcc.cc:205] Tried to move back new op lower bound from 7334420306075480064 to 7334420305826758656. Current Snapshot: MvccSnapshot[applied={T|T < 7334420306075480064 or (T in {7334420306075480064})}]
I20260928 21:12:37.544276 8448 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.544379 8448 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 09ea2943db314dbf8ed9bed70b51693e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.545483 8482 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.545578 8482 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P 703245b46b824f9a82bfd02dc02bbec9: This TxnStatusTablet replica's current role is: LEADER
I20260928 21:12:37.547591 8452 tablet_replica.cc:439] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } } }
I20260928 21:12:37.547709 8452 tablet_replica.cc:442] T c042f7071b4247939bbea194f7a8a651 P f2db6797d0c34a7a8f257a14b2e70f26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260928 21:12:37.558760 8492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.555020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46079 (local address 127.0.98.3:33051)
0928 21:12:37.555285 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.555299 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.555433 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.555754 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.555757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.555766 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:37.555868 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.555874 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.556935 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.556938 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.557559 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.557567 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.558111 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.558113 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.558171 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.558209 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.558524 (+ 315us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.558528 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.558577 (+ 49us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.558643 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260928 21:12:37.558795 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.555118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33051 (local address 127.0.98.2:46079)
0928 21:12:37.555264 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.555268 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.555381 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:12:37.555588 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.555592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.555654 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.555735 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:37.555990 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.556812 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.557667 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.558006 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.558110 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.558328 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.558417 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.558509 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.558612 (+ 103us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.558667 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 21:12:37.565451 8088 raft_consensus.cc:1261] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Refusing update from remote peer f2db6797d0c34a7a8f257a14b2e70f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.565474 8228 raft_consensus.cc:1261] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Refusing update from remote peer f2db6797d0c34a7a8f257a14b2e70f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.565819 8452 consensus_queue.cc:1035] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.565975 8449 consensus_queue.cc:1035] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.590252 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.586550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58969 (local address 127.0.98.3:43577)
0928 21:12:37.586672 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.586677 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.586880 (+ 203us) server_negotiation.cc:407] Connection header received
0928 21:12:37.586977 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.586981 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.587031 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.587163 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:37.587488 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.588168 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.589180 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.589403 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.589516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.589808 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.589885 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.589971 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.590062 (+ 91us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.590110 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:12:37.590286 8497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.586498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43577 (local address 127.0.98.2:58969)
0928 21:12:37.586799 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.586812 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:37.586896 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:37.587162 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:37.587164 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:37.587172 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0928 21:12:37.587322 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.587333 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.588310 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.588315 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.589055 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:37.589065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.589537 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.589541 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:37.589636 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.589676 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.589974 (+ 298us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.589983 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:12:37.590042 (+ 59us) client_negotiation.cc:240] Negotiation successful
0928 21:12:37.590118 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":110,"threads_started":1}
I20260928 21:12:37.614723 7930 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.614868 7930 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.617066 7930 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Sending DeleteTablet for 3 replicas of tablet b5ebf652c747447eae6593ea4fd4d3f1
I20260928 21:12:37.617661 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ebf652c747447eae6593ea4fd4d3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260928 21:12:37.617661 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ebf652c747447eae6593ea4fd4d3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60762
I20260928 21:12:37.617861 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet b5ebf652c747447eae6593ea4fd4d3f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60204
I20260928 21:12:37.618031 8499 tablet_replica.cc:330] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: stopping tablet replica
I20260928 21:12:37.618194 8501 tablet_replica.cc:330] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: stopping tablet replica
I20260928 21:12:37.618211 8499 raft_consensus.cc:2229] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.618312 8501 raft_consensus.cc:2229] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:37.618471 8499 raft_consensus.cc:2258] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.618597 8501 raft_consensus.cc:2258] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.618809 8500 tablet_replica.cc:330] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: stopping tablet replica
I20260928 21:12:37.618990 8500 raft_consensus.cc:2229] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.619154 8500 raft_consensus.cc:2258] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.619661 8501 ts_tablet_manager.cc:1905] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.619855 8500 ts_tablet_manager.cc:1905] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.622404 8499 ts_tablet_manager.cc:1905] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.622776 8500 ts_tablet_manager.cc:1918] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:37.622776 8501 ts_tablet_manager.cc:1918] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:37.622854 8501 log.cc:1196] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/b5ebf652c747447eae6593ea4fd4d3f1
I20260928 21:12:37.622967 8500 log.cc:1196] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/b5ebf652c747447eae6593ea4fd4d3f1
I20260928 21:12:37.623165 8501 ts_tablet_manager.cc:1939] T b5ebf652c747447eae6593ea4fd4d3f1 P f2db6797d0c34a7a8f257a14b2e70f26: Deleting consensus metadata
I20260928 21:12:37.623339 8500 ts_tablet_manager.cc:1939] T b5ebf652c747447eae6593ea4fd4d3f1 P 703245b46b824f9a82bfd02dc02bbec9: Deleting consensus metadata
I20260928 21:12:37.624027 7915 catalog_manager.cc:4841] TS 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079): tablet b5ebf652c747447eae6593ea4fd4d3f1 (table test-table [id=44d02bda7444446890039b137f84d29b]) successfully deleted
I20260928 21:12:37.624034 7918 catalog_manager.cc:4841] TS f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577): tablet b5ebf652c747447eae6593ea4fd4d3f1 (table test-table [id=44d02bda7444446890039b137f84d29b]) successfully deleted
I20260928 21:12:37.625346 8499 ts_tablet_manager.cc:1918] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:37.625427 8499 log.cc:1196] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/b5ebf652c747447eae6593ea4fd4d3f1
I20260928 21:12:37.625676 8499 ts_tablet_manager.cc:1939] T b5ebf652c747447eae6593ea4fd4d3f1 P 09ea2943db314dbf8ed9bed70b51693e: Deleting consensus metadata
I20260928 21:12:37.626381 7916 catalog_manager.cc:4841] TS 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273): tablet b5ebf652c747447eae6593ea4fd4d3f1 (table test-table [id=44d02bda7444446890039b137f84d29b]) successfully deleted
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[7886](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:12:37.649464 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:37.641256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35676 (local address 127.0.98.62:35255)
0928 21:12:37.641470 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.641474 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.641630 (+ 156us) server_negotiation.cc:407] Connection header received
0928 21:12:37.641785 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.641789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.641843 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.641926 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:37.642785 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.643589 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.644881 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.645267 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.645389 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.649318 (+ 3929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35676: BlockingRecv error: recv got EOF from 127.0.0.1:35676 (error 108)
Metrics: {"server-negotiator.queue_time_us":125}
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[7886](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629957, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260928 21:12:37.660662 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.651044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35680 (local address 127.0.98.62:35255)
0928 21:12:37.651167 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.651172 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.652029 (+ 857us) server_negotiation.cc:407] Connection header received
0928 21:12:37.652180 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.652183 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.652238 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.652334 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:37.653063 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.653911 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.654671 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.655007 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.655105 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.657436 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:37.657440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:37.657443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:37.657475 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:37.658988 (+ 1513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:37.659445 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:37.659455 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:37.659457 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:37.659542 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:37.659807 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:37.659810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:37.659812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:37.660018 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:37.660140 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.660332 (+ 192us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.660456 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:12:37.663407 7930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:12:37.663851 7930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:37.668438 8067 tablet_service.cc:1467] Processing CreateTablet for tablet b6354a940051415fa43d34b1e7af4c1f (DEFAULT_TABLE table=test-table [id=a7647cd64572499eb3fa20a47c584917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.668426 8208 tablet_service.cc:1467] Processing CreateTablet for tablet b6354a940051415fa43d34b1e7af4c1f (DEFAULT_TABLE table=test-table [id=a7647cd64572499eb3fa20a47c584917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.668766 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6354a940051415fa43d34b1e7af4c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.668772 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6354a940051415fa43d34b1e7af4c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.670689 8443 tablet_bootstrap.cc:492] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Bootstrap starting.
I20260928 21:12:37.670717 8444 tablet_bootstrap.cc:492] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Bootstrap starting.
I20260928 21:12:37.670969 8346 tablet_service.cc:1467] Processing CreateTablet for tablet b6354a940051415fa43d34b1e7af4c1f (DEFAULT_TABLE table=test-table [id=a7647cd64572499eb3fa20a47c584917]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.671241 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b6354a940051415fa43d34b1e7af4c1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.671526 8443 tablet_bootstrap.cc:654] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.671613 8444 tablet_bootstrap.cc:654] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.672463 8443 tablet_bootstrap.cc:492] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: No bootstrap required, opened a new log
I20260928 21:12:37.672534 8443 ts_tablet_manager.cc:1397] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:37.672955 8443 raft_consensus.cc:348] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.673051 8443 raft_consensus.cc:374] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.673074 8443 raft_consensus.cc:729] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.673219 8443 consensus_queue.cc:260] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.673476 8443 ts_tablet_manager.cc:1428] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.673516 8442 tablet_bootstrap.cc:492] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Bootstrap starting.
I20260928 21:12:37.674363 8442 tablet_bootstrap.cc:654] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.675297 8444 tablet_bootstrap.cc:492] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: No bootstrap required, opened a new log
I20260928 21:12:37.675383 8444 ts_tablet_manager.cc:1397] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260928 21:12:37.675585 8442 tablet_bootstrap.cc:492] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: No bootstrap required, opened a new log
I20260928 21:12:37.675652 8442 ts_tablet_manager.cc:1397] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:37.675835 8444 raft_consensus.cc:348] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.675937 8444 raft_consensus.cc:374] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.675971 8444 raft_consensus.cc:729] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea2943db314dbf8ed9bed70b51693e, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.676025 8442 raft_consensus.cc:348] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.676116 8442 raft_consensus.cc:374] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.676147 8442 raft_consensus.cc:729] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.676098 8444 consensus_queue.cc:260] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.676258 8442 consensus_queue.cc:260] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.676373 8444 ts_tablet_manager.cc:1428] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260928 21:12:37.676535 8442 ts_tablet_manager.cc:1428] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.751758 8482 raft_consensus.cc:482] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:37.751956 8482 raft_consensus.cc:504] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.752480 8482 leader_election.cc:288] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577)
I20260928 21:12:37.752944 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6354a940051415fa43d34b1e7af4c1f" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" is_pre_election: true
I20260928 21:12:37.753137 8368 raft_consensus.cc:2454] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 0.
I20260928 21:12:37.753314 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6354a940051415fa43d34b1e7af4c1f" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e" is_pre_election: true
I20260928 21:12:37.753440 8162 leader_election.cc:302] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.753487 8087 raft_consensus.cc:2454] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 0.
I20260928 21:12:37.753625 8482 raft_consensus.cc:2781] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:37.753721 8482 raft_consensus.cc:482] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:37.753772 8482 raft_consensus.cc:3037] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.754753 8482 raft_consensus.cc:504] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.755087 8482 leader_election.cc:288] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 election: Requested vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577)
I20260928 21:12:37.755404 8368 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6354a940051415fa43d34b1e7af4c1f" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2db6797d0c34a7a8f257a14b2e70f26"
I20260928 21:12:37.755455 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b6354a940051415fa43d34b1e7af4c1f" candidate_uuid: "703245b46b824f9a82bfd02dc02bbec9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e"
I20260928 21:12:37.755505 8368 raft_consensus.cc:3037] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.755596 8087 raft_consensus.cc:3037] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.756455 8368 raft_consensus.cc:2454] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 1.
I20260928 21:12:37.756568 8087 raft_consensus.cc:2454] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 703245b46b824f9a82bfd02dc02bbec9 in term 1.
I20260928 21:12:37.756757 8162 leader_election.cc:302] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.756978 8482 raft_consensus.cc:2781] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:37.757076 8482 raft_consensus.cc:686] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 1 LEADER]: Becoming Leader. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Running, Role: LEADER
I20260928 21:12:37.757242 8482 consensus_queue.cc:237] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.759178 7931 catalog_manager.cc:5495] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "703245b46b824f9a82bfd02dc02bbec9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } health_report { overall_health: UNKNOWN } } }
I20260928 21:12:37.775156 8487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.771951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60778 (local address 127.0.98.2:46079)
0928 21:12:37.772090 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.772095 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.772110 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:37.772228 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.772230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.772276 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.772356 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:37.772611 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.773256 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.774113 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.774343 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.774458 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.774711 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.774783 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.774864 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.774954 (+ 90us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.775009 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:12:37.777478 8087 raft_consensus.cc:1261] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Refusing update from remote peer 703245b46b824f9a82bfd02dc02bbec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.777503 8368 raft_consensus.cc:1261] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Refusing update from remote peer 703245b46b824f9a82bfd02dc02bbec9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.777897 8450 consensus_queue.cc:1035] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.778060 8482 consensus_queue.cc:1035] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.788156 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.788292 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.790151 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Sending DeleteTablet for 3 replicas of tablet b6354a940051415fa43d34b1e7af4c1f
I20260928 21:12:37.790624 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet b6354a940051415fa43d34b1e7af4c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260928 21:12:37.790714 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet b6354a940051415fa43d34b1e7af4c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60762
I20260928 21:12:37.790828 8499 tablet_replica.cc:330] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: stopping tablet replica
I20260928 21:12:37.790882 8500 tablet_replica.cc:330] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: stopping tablet replica
I20260928 21:12:37.790944 8499 raft_consensus.cc:2229] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.790992 8500 raft_consensus.cc:2229] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:37.791066 8499 raft_consensus.cc:2258] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.791308 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet b6354a940051415fa43d34b1e7af4c1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60204
I20260928 21:12:37.791513 8501 tablet_replica.cc:330] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: stopping tablet replica
I20260928 21:12:37.791659 8501 raft_consensus.cc:2229] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.791802 8501 raft_consensus.cc:2258] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.791852 8500 raft_consensus.cc:2258] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.792234 7929 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260928 21:12:37.792444 8499 ts_tablet_manager.cc:1905] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:12:37.792636 7929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:12:37.792668 8500 ts_tablet_manager.cc:1905] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.795018 8499 ts_tablet_manager.cc:1918] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:37.795094 8500 ts_tablet_manager.cc:1918] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:37.795115 8499 log.cc:1196] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/b6354a940051415fa43d34b1e7af4c1f
I20260928 21:12:37.795171 8500 log.cc:1196] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/b6354a940051415fa43d34b1e7af4c1f
I20260928 21:12:37.795424 8500 ts_tablet_manager.cc:1939] T b6354a940051415fa43d34b1e7af4c1f P 703245b46b824f9a82bfd02dc02bbec9: Deleting consensus metadata
I20260928 21:12:37.796067 7915 catalog_manager.cc:4841] TS 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079): tablet b6354a940051415fa43d34b1e7af4c1f (table test-table [id=a7647cd64572499eb3fa20a47c584917]) successfully deleted
I20260928 21:12:37.796176 8499 ts_tablet_manager.cc:1939] T b6354a940051415fa43d34b1e7af4c1f P 09ea2943db314dbf8ed9bed70b51693e: Deleting consensus metadata
I20260928 21:12:37.796739 8501 ts_tablet_manager.cc:1905] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.797052 7916 catalog_manager.cc:4841] TS 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273): tablet b6354a940051415fa43d34b1e7af4c1f (table test-table [id=a7647cd64572499eb3fa20a47c584917]) successfully deleted
I20260928 21:12:37.798997 8067 tablet_service.cc:1467] Processing CreateTablet for tablet 407d1789611143538b1eda4fd177ea1c (DEFAULT_TABLE table=test-table [id=d8abafa3fa3b4d41b5fa59f42405a081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.799160 8501 ts_tablet_manager.cc:1918] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:12:37.799228 8501 log.cc:1196] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/b6354a940051415fa43d34b1e7af4c1f
I20260928 21:12:37.799258 8067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407d1789611143538b1eda4fd177ea1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.799333 8346 tablet_service.cc:1467] Processing CreateTablet for tablet 407d1789611143538b1eda4fd177ea1c (DEFAULT_TABLE table=test-table [id=d8abafa3fa3b4d41b5fa59f42405a081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.799477 8501 ts_tablet_manager.cc:1939] T b6354a940051415fa43d34b1e7af4c1f P f2db6797d0c34a7a8f257a14b2e70f26: Deleting consensus metadata
I20260928 21:12:37.799552 8346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407d1789611143538b1eda4fd177ea1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.799919 8208 tablet_service.cc:1467] Processing CreateTablet for tablet 407d1789611143538b1eda4fd177ea1c (DEFAULT_TABLE table=test-table [id=d8abafa3fa3b4d41b5fa59f42405a081]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:12:37.800141 8208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 407d1789611143538b1eda4fd177ea1c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:37.801772 8442 tablet_bootstrap.cc:492] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Bootstrap starting.
I20260928 21:12:37.802279 8443 tablet_bootstrap.cc:492] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: Bootstrap starting.
I20260928 21:12:37.802589 8442 tablet_bootstrap.cc:654] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.803061 8443 tablet_bootstrap.cc:654] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.803097 8444 tablet_bootstrap.cc:492] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e: Bootstrap starting.
I20260928 21:12:37.803646 7918 catalog_manager.cc:4841] TS f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577): tablet b6354a940051415fa43d34b1e7af4c1f (table test-table [id=a7647cd64572499eb3fa20a47c584917]) successfully deleted
I20260928 21:12:37.804003 8444 tablet_bootstrap.cc:654] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:37.804188 8443 tablet_bootstrap.cc:492] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: No bootstrap required, opened a new log
I20260928 21:12:37.804257 8443 ts_tablet_manager.cc:1397] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260928 21:12:37.804672 8443 raft_consensus.cc:348] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.804786 8443 raft_consensus.cc:374] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.804821 8443 raft_consensus.cc:729] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 703245b46b824f9a82bfd02dc02bbec9, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.804972 8443 consensus_queue.cc:260] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.805250 8443 ts_tablet_manager.cc:1428] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:12:37.806092 8444 tablet_bootstrap.cc:492] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e: No bootstrap required, opened a new log
I20260928 21:12:37.806181 8444 ts_tablet_manager.cc:1397] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260928 21:12:37.806635 8444 raft_consensus.cc:348] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.806778 8444 raft_consensus.cc:374] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.806813 8444 raft_consensus.cc:729] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09ea2943db314dbf8ed9bed70b51693e, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.806927 8444 consensus_queue.cc:260] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.807195 8444 ts_tablet_manager.cc:1428] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.808094 8442 tablet_bootstrap.cc:492] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: No bootstrap required, opened a new log
I20260928 21:12:37.808192 8442 ts_tablet_manager.cc:1397] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260928 21:12:37.808605 8442 raft_consensus.cc:348] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.808732 8442 raft_consensus.cc:374] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:37.808766 8442 raft_consensus.cc:729] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Initialized, Role: FOLLOWER
I20260928 21:12:37.808909 8442 consensus_queue.cc:260] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.809201 8442 ts_tablet_manager.cc:1428] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:12:37.815647 8495 raft_consensus.cc:482] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:12:37.815743 8495 raft_consensus.cc:504] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.816104 8495 leader_election.cc:288] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079)
I20260928 21:12:37.816407 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407d1789611143538b1eda4fd177ea1c" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703245b46b824f9a82bfd02dc02bbec9" is_pre_election: true
I20260928 21:12:37.816411 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407d1789611143538b1eda4fd177ea1c" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e" is_pre_election: true
I20260928 21:12:37.816514 8228 raft_consensus.cc:2454] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 0.
I20260928 21:12:37.816534 8087 raft_consensus.cc:2454] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 0.
I20260928 21:12:37.816862 8299 leader_election.cc:302] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09ea2943db314dbf8ed9bed70b51693e, 703245b46b824f9a82bfd02dc02bbec9, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.817086 8495 raft_consensus.cc:2781] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:12:37.817152 8495 raft_consensus.cc:482] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:12:37.817191 8495 raft_consensus.cc:3037] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.818308 8495 raft_consensus.cc:504] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.818651 8495 leader_election.cc:288] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 election: Requested vote from peers 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273), 703245b46b824f9a82bfd02dc02bbec9 (127.0.98.2:46079)
I20260928 21:12:37.818949 8087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407d1789611143538b1eda4fd177ea1c" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09ea2943db314dbf8ed9bed70b51693e"
I20260928 21:12:37.818962 8228 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "407d1789611143538b1eda4fd177ea1c" candidate_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "703245b46b824f9a82bfd02dc02bbec9"
I20260928 21:12:37.819072 8228 raft_consensus.cc:3037] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.819072 8087 raft_consensus.cc:3037] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:37.820179 8228 raft_consensus.cc:2454] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 1.
I20260928 21:12:37.820179 8087 raft_consensus.cc:2454] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2db6797d0c34a7a8f257a14b2e70f26 in term 1.
I20260928 21:12:37.820502 8300 leader_election.cc:302] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09ea2943db314dbf8ed9bed70b51693e, f2db6797d0c34a7a8f257a14b2e70f26; no voters:
I20260928 21:12:37.820674 8495 raft_consensus.cc:2781] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:37.820760 8495 raft_consensus.cc:686] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 LEADER]: Becoming Leader. State: Replica: f2db6797d0c34a7a8f257a14b2e70f26, State: Running, Role: LEADER
I20260928 21:12:37.820921 8495 consensus_queue.cc:237] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } }
I20260928 21:12:37.822431 7931 catalog_manager.cc:5495] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 reported cstate change: term changed from 0 to 1, leader changed from <none> to f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2db6797d0c34a7a8f257a14b2e70f26" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43577 } health_report { overall_health: HEALTHY } } }
I20260928 21:12:37.838449 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.835184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60210 (local address 127.0.98.3:43577)
0928 21:12:37.835291 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:37.835296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:37.835313 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:12:37.835478 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:37.835480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:37.835524 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:37.835586 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:12:37.835825 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.836318 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.837475 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:37.837682 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:37.837760 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:37.838022 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:12:37.838086 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:12:37.838138 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:37.838249 (+ 111us) server_negotiation.cc:299] Negotiation successful
0928 21:12:37.838291 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:12:37.844815 8087 raft_consensus.cc:1261] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Refusing update from remote peer f2db6797d0c34a7a8f257a14b2e70f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.844929 8228 raft_consensus.cc:1261] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Refusing update from remote peer f2db6797d0c34a7a8f257a14b2e70f26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:12:37.845232 8495 consensus_queue.cc:1035] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09ea2943db314dbf8ed9bed70b51693e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.845376 8449 consensus_queue.cc:1035] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "703245b46b824f9a82bfd02dc02bbec9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:12:37.884814 7929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.885030 7929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:12:37.887153 7929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 37024f3e813b4e14b4d8e41d52243a98: Sending DeleteTablet for 3 replicas of tablet 407d1789611143538b1eda4fd177ea1c
I20260928 21:12:37.887627 8067 tablet_service.cc:1514] Processing DeleteTablet for tablet 407d1789611143538b1eda4fd177ea1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54270
I20260928 21:12:37.887662 8208 tablet_service.cc:1514] Processing DeleteTablet for tablet 407d1789611143538b1eda4fd177ea1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60762
I20260928 21:12:37.887821 8499 tablet_replica.cc:330] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e: stopping tablet replica
I20260928 21:12:37.887846 8500 tablet_replica.cc:330] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: stopping tablet replica
I20260928 21:12:37.887946 8499 raft_consensus.cc:2229] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.887965 8500 raft_consensus.cc:2229] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:12:37.888067 8500 raft_consensus.cc:2258] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.888082 8499 raft_consensus.cc:2258] T 407d1789611143538b1eda4fd177ea1c P 09ea2943db314dbf8ed9bed70b51693e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.888523 8346 tablet_service.cc:1514] Processing DeleteTablet for tablet 407d1789611143538b1eda4fd177ea1c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:12:37 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60204
I20260928 21:12:37.888671 8501 tablet_replica.cc:330] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: stopping tablet replica
I20260928 21:12:37.888765 8501 raft_consensus.cc:2229] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:12:37.888952 8501 raft_consensus.cc:2258] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:12:37.889568 8501 ts_tablet_manager.cc:1905] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.889755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8003
I20260928 21:12:37.892637 8501 ts_tablet_manager.cc:1918] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260928 21:12:37.893083 8501 log.cc:1196] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/407d1789611143538b1eda4fd177ea1c
I20260928 21:12:37.893427 8501 ts_tablet_manager.cc:1939] T 407d1789611143538b1eda4fd177ea1c P f2db6797d0c34a7a8f257a14b2e70f26: Deleting consensus metadata
I20260928 21:12:37.898901 7918 catalog_manager.cc:4841] TS f2db6797d0c34a7a8f257a14b2e70f26 (127.0.98.3:43577): tablet 407d1789611143538b1eda4fd177ea1c (table test-table [id=d8abafa3fa3b4d41b5fa59f42405a081]) successfully deleted
I20260928 21:12:37.899530 8500 ts_tablet_manager.cc:1905] T 407d1789611143538b1eda4fd177ea1c P 703245b46b824f9a82bfd02dc02bbec9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:12:37.899549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8143
W20260928 21:12:37.900765 7916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34273 (error 108)
I20260928 21:12:37.901881 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.901504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34273 (local address 127.0.0.1:54284)
0928 21:12:37.901714 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.901787 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":50,"threads_started":1}
W20260928 21:12:37.902016 7916 catalog_manager.cc:4584] TS 09ea2943db314dbf8ed9bed70b51693e (127.0.98.1:34273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 407d1789611143538b1eda4fd177ea1c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34273: connect: Connection refused (error 111)
I20260928 21:12:37.906193 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8283
I20260928 21:12:37.907173 8530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:37.906077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46079 (local address 127.0.0.1:60790)
0928 21:12:37.906251 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:37.907082 (+ 831us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:46079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 21:12:37.911949 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 7899
2026-09-28T21:12:37Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3143 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[8533](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[8533](info): set up 2 sockets
Sep 28 21:12:37 dist-test-slave-v7xz krb5kdc[8533](info): commencing operation
krb5kdc: starting...
W20260928 21:12:38.957610 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:39 dist-test-slave-v7xz krb5kdc[8533](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629959, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:39.082731 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33443
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:39.198194 8546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:39.198550 8546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:39.198661 8546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:39.202930 8546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:39.203037 8546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:39.203076 8546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:39.203135 8546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:39.203183 8546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:39.208981 8546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33443
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:39.210158 8546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:39.211249 8546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:39.219952 8551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:39.220070 8546 server_base.cc:1034] running on GCE node
W20260928 21:12:39.219980 8554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:39.219952 8552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:39.220820 8546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:39.221942 8546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:39.223094 8546 hybrid_clock.cc:648] HybridClock initialized: now 1790629959223079 us; error 28 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-28T21:12:39Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1317 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:39 dist-test-slave-v7xz krb5kdc[8559](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:39 dist-test-slave-v7xz krb5kdc[8559](info): set up 2 sockets
Sep 28 21:12:39 dist-test-slave-v7xz krb5kdc[8559](info): commencing operation
krb5kdc: starting...
W20260928 21:12:40.277661 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:12:40 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:12:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:12:40.401866 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38079
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:40.512398 8572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:40.512691 8572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:40.512732 8572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:40.516752 8572 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:40.516844 8572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:40.516865 8572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:40.516934 8572 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:40.516961 8572 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:40.522289 8572 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:40.523396 8572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:40.524461 8572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:40.532876 8580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:40.532917 8578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:40.532902 8577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:40.533053 8572 server_base.cc:1034] running on GCE node
I20260928 21:12:40.533704 8572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:40.534868 8572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:40.536026 8572 hybrid_clock.cc:648] HybridClock initialized: now 1790629960535985 us; error 56 us; skew 500 ppm
Sep 28 21:12:40 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:40.538995 8572 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:40.539947 8572 webserver.cc:460] Webserver started at http://127.0.98.62:43089/ using document root <none> and password file <none>
I20260928 21:12:40.540503 8572 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:40.540565 8572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:40.540799 8572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:40.542847 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "92b01a20b85d4bae98957168292bac7c"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "c56e5727af39834fa4806fab58bd4143"
server_key_iv: "cdfddd597a3bdc8485e3ed21ea0c8aa3"
server_key_version: "encryptionkey@0"
I20260928 21:12:40.543412 8572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "92b01a20b85d4bae98957168292bac7c"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "c56e5727af39834fa4806fab58bd4143"
server_key_iv: "cdfddd597a3bdc8485e3ed21ea0c8aa3"
server_key_version: "encryptionkey@0"
I20260928 21:12:40.547032 8572 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 21:12:40.549460 8587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:40.550724 8572 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:12:40.550844 8572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "92b01a20b85d4bae98957168292bac7c"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "c56e5727af39834fa4806fab58bd4143"
server_key_iv: "cdfddd597a3bdc8485e3ed21ea0c8aa3"
server_key_version: "encryptionkey@0"
I20260928 21:12:40.550951 8572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:40.588507 8572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:40.591780 8572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:40.592011 8572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:40.599649 8572 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38079
I20260928 21:12:40.599658 8639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38079 every 8 connection(s)
I20260928 21:12:40.600660 8572 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:40.603720 8640 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:40.608619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8572
I20260928 21:12:40.608783 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:40.609088 392 external_mini_cluster.cc:1428] Setting key ef447d0d8513a9658eaa458172976b69
I20260928 21:12:40.609809 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Bootstrap starting.
I20260928 21:12:40.612250 8640 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:40.613358 8640 log.cc:824] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:40.615782 8640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: No bootstrap required, opened a new log
Sep 28 21:12:40 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629960, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:40.620581 8640 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:40.620878 8640 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:40.620985 8640 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b01a20b85d4bae98957168292bac7c, State: Initialized, Role: FOLLOWER
I20260928 21:12:40.621481 8640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:40.621647 8640 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:40.621753 8640 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:40.621872 8640 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:40.623061 8640 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:40.623502 8640 leader_election.cc:302] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b01a20b85d4bae98957168292bac7c; no voters:
I20260928 21:12:40.623924 8640 leader_election.cc:288] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:40.624011 8645 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:40.624250 8645 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 LEADER]: Becoming Leader. State: Replica: 92b01a20b85d4bae98957168292bac7c, State: Running, Role: LEADER
I20260928 21:12:40.624567 8645 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:40.624943 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:40.610564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57124 (local address 127.0.98.62:38079)
0928 21:12:40.610988 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:40.611003 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:40.611039 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:12:40.611701 (+ 662us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:40.611718 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:40.612065 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:40.612386 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:40.613412 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:40.614641 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:40.615940 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:40.616469 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:40.616610 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:40.619583 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:40.619588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:40.619603 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:40.619631 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:40.621602 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:40.622217 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:40.622222 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:40.622226 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:40.622340 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:40.622637 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:40.622639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:40.622640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:40.623023 (+ 383us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:40.623237 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:40.623522 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 21:12:40.623840 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":140,"threads_started":1}
I20260928 21:12:40.625458 8640 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:40.627071 8647 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b01a20b85d4bae98957168292bac7c. Latest consensus state: current_term: 1 leader_uuid: "92b01a20b85d4bae98957168292bac7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } } }
I20260928 21:12:40.627213 8647 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:40.627588 8657 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:40.627988 8646 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92b01a20b85d4bae98957168292bac7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } } }
I20260928 21:12:40.628204 8646 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:40.630662 8657 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:40.636262 8657 catalog_manager.cc:1347] Generated new cluster ID: c646e5686e3a4f66b8623582b7c152bb
I20260928 21:12:40.636333 8657 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:40.663805 8657 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:40.664880 8657 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:40.673097 8657 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Generated new TSK 0
I20260928 21:12:40.673969 8657 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:12:40.712630 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:40.825712 8666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:40.826030 8666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:40.826118 8666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:40.830516 8666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:40.830585 8666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:40.830693 8666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:40.836118 8666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:40.837251 8666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:40.838356 8666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:40.848145 8674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:40.848193 8671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:40.848145 8672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:40.848435 8666 server_base.cc:1034] running on GCE node
I20260928 21:12:40.849058 8666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:40.849715 8666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:40.850891 8666 hybrid_clock.cc:648] HybridClock initialized: now 1790629960850853 us; error 49 us; skew 500 ppm
Sep 28 21:12:40 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:40.853852 8666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:12:40.854854 8666 webserver.cc:460] Webserver started at http://127.0.98.1:34229/ using document root <none> and password file <none>
I20260928 21:12:40.855458 8666 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:40.855553 8666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:40.855774 8666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:40.857936 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "b0c0714672a9478e9908464346f39591"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "0389349cd6c3e302b85f0aa35d9d9443"
server_key_iv: "3bac14a336203bff08ba9f10349e732d"
server_key_version: "encryptionkey@0"
I20260928 21:12:40.858570 8666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b0c0714672a9478e9908464346f39591"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "0389349cd6c3e302b85f0aa35d9d9443"
server_key_iv: "3bac14a336203bff08ba9f10349e732d"
server_key_version: "encryptionkey@0"
I20260928 21:12:40.862323 8666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:40.864796 8681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:40.866118 8666 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:40.866215 8666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "b0c0714672a9478e9908464346f39591"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "0389349cd6c3e302b85f0aa35d9d9443"
server_key_iv: "3bac14a336203bff08ba9f10349e732d"
server_key_version: "encryptionkey@0"
I20260928 21:12:40.866353 8666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:40.885015 8666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:40.887806 8666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:40.888002 8666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:40.888615 8666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:40.889497 8666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:40.889542 8666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:40.889611 8666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:40.889642 8666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:40.898413 8666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40561
I20260928 21:12:40.898494 8794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40561 every 8 connection(s)
I20260928 21:12:40.899418 8666 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
Sep 28 21:12:40 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629960, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:40.909391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8666
I20260928 21:12:40.909523 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:40.909859 392 external_mini_cluster.cc:1428] Setting key 29a31eb6fce9c9289275208977b7be69
I20260928 21:12:40.913647 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:40.901372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52343 (local address 127.0.98.62:38079)
0928 21:12:40.901513 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:40.901517 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:40.902371 (+ 854us) server_negotiation.cc:407] Connection header received
0928 21:12:40.903791 (+ 1420us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:40.903794 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:40.903848 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:40.903928 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:40.905130 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:40.905671 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:40.906859 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:40.907107 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:40.907222 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:40.910011 (+ 2789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:40.910019 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:40.910021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:40.910050 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:40.911592 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:40.912084 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:40.912090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:40.912092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:40.912178 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:40.912534 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:40.912537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:40.912539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:40.912716 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:40.912821 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:40.913359 (+ 538us) server_negotiation.cc:299] Negotiation successful
0928 21:12:40.913477 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:12:40.914515 8797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:40.901783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:52343)
0928 21:12:40.902217 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:40.902259 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:40.903580 (+ 1321us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:40.904097 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:40.904104 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:40.904458 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:40.904976 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:40.904992 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:40.905801 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:40.905804 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:40.906727 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:40.906736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:40.907265 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:40.907269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:40.907459 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:40.908049 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:40.908064 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:40.909809 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:40.911701 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:40.911707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:40.911723 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:40.911983 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:40.912270 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:40.912274 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:40.912277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:40.912418 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:40.912854 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:40.912863 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:40.913125 (+ 262us) client_negotiation.cc:769] Sending connection context
0928 21:12:40.913322 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:12:40.913667 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":102,"threads_started":1}
I20260928 21:12:40.915817 8795 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38079
I20260928 21:12:40.916095 8795 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:40.916756 8795 heartbeater.cc:511] Master 127.0.98.62:38079 requested a full tablet report, sending...
I20260928 21:12:40.918733 8604 ts_manager.cc:194] Registered new tserver with Master: b0c0714672a9478e9908464346f39591 (127.0.98.1:40561)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:12:40.920444 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52343
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:12:40.939441 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:41.052695 8800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:41.053093 8800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:41.053176 8800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:41.057632 8800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:41.057700 8800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:41.057791 8800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:41.063403 8800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:41.064625 8800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:41.065771 8800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:41.075665 8806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.075670 8805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.075670 8808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.075973 8800 server_base.cc:1034] running on GCE node
I20260928 21:12:41.076499 8800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:41.077229 8800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:41.078419 8800 hybrid_clock.cc:648] HybridClock initialized: now 1790629961078389 us; error 55 us; skew 500 ppm
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:41.081359 8800 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:12:41.082309 8800 webserver.cc:460] Webserver started at http://127.0.98.2:38423/ using document root <none> and password file <none>
I20260928 21:12:41.082940 8800 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:41.083042 8800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:41.083302 8800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:41.085626 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "6cac52a2798c47bd8b54dfc13ea5a119"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "e76e6151a39a02dfbeeb383a6b16e7f2"
server_key_iv: "c64ef7d11796925898721ec56bcf0af9"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.086238 8800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6cac52a2798c47bd8b54dfc13ea5a119"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "e76e6151a39a02dfbeeb383a6b16e7f2"
server_key_iv: "c64ef7d11796925898721ec56bcf0af9"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.090214 8800 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:41.093003 8815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.094487 8800 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:12:41.094624 8800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "6cac52a2798c47bd8b54dfc13ea5a119"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "e76e6151a39a02dfbeeb383a6b16e7f2"
server_key_iv: "c64ef7d11796925898721ec56bcf0af9"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.094734 8800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:41.105368 8800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:41.108172 8800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:41.108322 8800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:41.108908 8800 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:41.109707 8800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:41.109753 8800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.109793 8800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:41.109815 8800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.118819 8800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42733
I20260928 21:12:41.118896 8928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42733 every 8 connection(s)
I20260928 21:12:41.119776 8800 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:41.126184 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8800
I20260928 21:12:41.126332 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:41.126688 392 external_mini_cluster.cc:1428] Setting key cd444b7b89b028f594c11210413ccdd8
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:41.135567 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.121734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35861 (local address 127.0.98.62:38079)
0928 21:12:41.121870 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:41.121874 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:41.122808 (+ 934us) server_negotiation.cc:407] Connection header received
0928 21:12:41.124268 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:41.124273 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:41.124342 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:41.124437 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:41.125840 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.126546 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.128134 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.128429 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.128557 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.131740 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:41.131748 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:41.131754 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:41.131789 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:41.133428 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:41.133982 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:41.133985 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:41.133987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:41.134072 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:41.134443 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:41.134446 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:41.134447 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:41.134634 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:41.134746 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:41.135270 (+ 524us) server_negotiation.cc:299] Negotiation successful
0928 21:12:41.135402 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:12:41.136500 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.122197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:35861)
0928 21:12:41.122637 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:41.122681 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:41.124011 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:41.124604 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:41.124610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:41.125115 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:41.125681 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.125703 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.126710 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.126714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.128011 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.128023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.128614 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.128621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.128883 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.129530 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:41.129558 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:41.131566 (+ 2008us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:41.133557 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:41.133564 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:41.133582 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:41.133872 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:41.134175 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:41.134178 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:41.134180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:41.134337 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:41.134756 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:41.134764 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:41.135022 (+ 258us) client_negotiation.cc:769] Sending connection context
0928 21:12:41.135224 (+ 202us) client_negotiation.cc:240] Negotiation successful
0928 21:12:41.135572 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260928 21:12:41.137777 8929 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38079
I20260928 21:12:41.138041 8929 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:41.138626 8929 heartbeater.cc:511] Master 127.0.98.62:38079 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:12:41.139837 8604 ts_manager.cc:194] Registered new tserver with Master: 6cac52a2798c47bd8b54dfc13ea5a119 (127.0.98.2:42733)
I20260928 21:12:41.140733 8604 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35861
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:12:41.156719 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:41.271343 8934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:41.271689 8934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:41.271783 8934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:41.276111 8934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:41.276191 8934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:41.276254 8934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:41.281754 8934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8934
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:41.282852 8934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:41.283859 8934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:41.293798 8940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.293872 8942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.293889 8939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.293943 8934 server_base.cc:1034] running on GCE node
I20260928 21:12:41.294525 8934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:41.295187 8934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:41.296371 8934 hybrid_clock.cc:648] HybridClock initialized: now 1790629961296321 us; error 70 us; skew 500 ppm
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:41.299677 8934 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:12:41.300585 8934 webserver.cc:460] Webserver started at http://127.0.98.3:33659/ using document root <none> and password file <none>
I20260928 21:12:41.301239 8934 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:41.301334 8934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:41.301543 8934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:41.303674 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "039c8e74b286470c810f7742197eb562"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "4ce648d3ed8501e5a151c5d550dc9466"
server_key_iv: "bbb0ce9e35c539a4d7da932c54a808db"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.304255 8934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "039c8e74b286470c810f7742197eb562"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "4ce648d3ed8501e5a151c5d550dc9466"
server_key_iv: "bbb0ce9e35c539a4d7da932c54a808db"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.308053 8934 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 21:12:41.310662 8949 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.311854 8934 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:41.311995 8934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "039c8e74b286470c810f7742197eb562"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "4ce648d3ed8501e5a151c5d550dc9466"
server_key_iv: "bbb0ce9e35c539a4d7da932c54a808db"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.312109 8934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:41.342574 8934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:41.345757 8934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:41.345978 8934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:41.346593 8934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:41.347438 8934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:41.347513 8934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.347585 8934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:41.347636 8934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.357370 8934 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45005
I20260928 21:12:41.357388 9062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45005 every 8 connection(s)
I20260928 21:12:41.358603 8934 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:41.363818 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 8934
I20260928 21:12:41.363981 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:41.364284 392 external_mini_cluster.cc:1428] Setting key 66cc62f9c7af2bcf8b7befff7af6be4c
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:12:41.374349 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.360946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50317 (local address 127.0.98.62:38079)
0928 21:12:41.361132 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:41.361137 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:41.362297 (+ 1160us) server_negotiation.cc:407] Connection header received
0928 21:12:41.363757 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:41.363761 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:41.363832 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:41.363952 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:41.365802 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.366421 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.367643 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.367855 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.367941 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.370566 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:41.370569 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:41.370571 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:41.370603 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:41.372193 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:41.372738 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:41.372742 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:41.372744 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:41.372875 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:41.373223 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:41.373227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:41.373228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:41.373426 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:41.373538 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:41.374061 (+ 523us) server_negotiation.cc:299] Negotiation successful
0928 21:12:41.374186 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:12:41.375208 9065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.361533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:50317)
0928 21:12:41.362116 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:41.362164 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:41.363490 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:41.364181 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:41.364221 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:41.364815 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:41.365618 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.365640 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.366563 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.366566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.367524 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.367533 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.367991 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.367994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.368136 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.368721 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:41.368737 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:41.370406 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:41.372331 (+ 1925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:41.372337 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:41.372354 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:41.372627 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:41.372963 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:12:41.372966 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:12:41.372968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:12:41.373120 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:12:41.373536 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:41.373543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:41.373812 (+ 269us) client_negotiation.cc:769] Sending connection context
0928 21:12:41.374010 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 21:12:41.374363 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":155,"threads_started":1}
I20260928 21:12:41.376390 9063 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38079
I20260928 21:12:41.376659 9063 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:41.377351 9063 heartbeater.cc:511] Master 127.0.98.62:38079 requested a full tablet report, sending...
I20260928 21:12:41.378449 8602 ts_manager.cc:194] Registered new tserver with Master: 039c8e74b286470c810f7742197eb562 (127.0.98.3:45005)
I20260928 21:12:41.379042 8602 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:50317
I20260928 21:12:41.390482 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260928 21:12:41.467024 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8666
I20260928 21:12:41.472582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8800
I20260928 21:12:41.477833 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8934
I20260928 21:12:41.483038 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 8572
I20260928 21:12:41.489225 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38079
--webserver_interface=127.0.98.62
--webserver_port=43089
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38079
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:12:41.601974 9072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:41.602310 9072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:41.602393 9072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:41.606472 9072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:41.606570 9072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:41.606608 9072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:41.606647 9072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:41.606675 9072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:41.612004 9072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38079
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=43089
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:41.613134 9072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:41.614159 9072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:41.622495 9080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.622545 9072 server_base.cc:1034] running on GCE node
W20260928 21:12:41.622426 9077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.622426 9078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.623250 9072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:41.624318 9072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:41.625473 9072 hybrid_clock.cc:648] HybridClock initialized: now 1790629961625459 us; error 27 us; skew 500 ppm
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:41.628252 9072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:12:41.629273 9072 webserver.cc:460] Webserver started at http://127.0.98.62:43089/ using document root <none> and password file <none>
I20260928 21:12:41.629848 9072 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:41.629956 9072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:41.633841 9072 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:41.636010 9087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.637362 9072 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:41.637522 9072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "92b01a20b85d4bae98957168292bac7c"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "c56e5727af39834fa4806fab58bd4143"
server_key_iv: "cdfddd597a3bdc8485e3ed21ea0c8aa3"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.638068 9072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:41.649479 9072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:41.652509 9072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:41.652743 9072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:41.660382 9072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38079
I20260928 21:12:41.660400 9139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38079 every 8 connection(s)
I20260928 21:12:41.661573 9072 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:41.665380 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9072
I20260928 21:12:41.665827 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:40561
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38079
--webserver_port=34229
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:12:41.665979 9140 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:12:41.667627 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Bootstrap starting.
I20260928 21:12:41.683924 9140 log.cc:824] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:41.690470 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:12:41.690860 9140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Bootstrap complete.
I20260928 21:12:41.695017 9140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:41.695422 9140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b01a20b85d4bae98957168292bac7c, State: Initialized, Role: FOLLOWER
I20260928 21:12:41.695951 9140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:41.696103 9140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:41.696190 9140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:41.696309 9140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:12:41.698271 9140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:41.698608 9140 leader_election.cc:302] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b01a20b85d4bae98957168292bac7c; no voters:
I20260928 21:12:41.698967 9140 leader_election.cc:288] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:12:41.699040 9144 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:12:41.699255 9144 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [term 2 LEADER]: Becoming Leader. State: Replica: 92b01a20b85d4bae98957168292bac7c, State: Running, Role: LEADER
I20260928 21:12:41.699549 9144 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } }
I20260928 21:12:41.700258 9140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:41.700973 9145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "92b01a20b85d4bae98957168292bac7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } } }
I20260928 21:12:41.701118 9145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:41.701431 9146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92b01a20b85d4bae98957168292bac7c. Latest consensus state: current_term: 2 leader_uuid: "92b01a20b85d4bae98957168292bac7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b01a20b85d4bae98957168292bac7c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38079 } } }
I20260928 21:12:41.701509 9146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:41.701924 9151 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:41.705030 9151 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:41.705636 9151 catalog_manager.cc:1259] Loaded cluster ID: c646e5686e3a4f66b8623582b7c152bb
I20260928 21:12:41.705696 9151 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:41.708045 9151 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:41.708696 9151 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 92b01a20b85d4bae98957168292bac7c: Loaded TSK: 0
I20260928 21:12:41.709910 9151 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:12:41.781396 9142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:41.781704 9142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:41.781790 9142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:41.786020 9142 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:41.786123 9142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:41.786216 9142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:41.791755 9142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:40561
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34229
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9142
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:41.792812 9142 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:41.793895 9142 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:41.804730 9168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.804885 9142 server_base.cc:1034] running on GCE node
W20260928 21:12:41.805074 9167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.804705 9170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.805552 9142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:41.806216 9142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:41.807403 9142 hybrid_clock.cc:648] HybridClock initialized: now 1790629961807376 us; error 46 us; skew 500 ppm
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:41.810156 9142 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260928 21:12:41.811102 9142 webserver.cc:460] Webserver started at http://127.0.98.1:34229/ using document root <none> and password file <none>
I20260928 21:12:41.811712 9142 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:41.811784 9142 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:41.815629 9142 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.001s
I20260928 21:12:41.817699 9177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.818976 9142 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:12:41.819067 9142 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "b0c0714672a9478e9908464346f39591"
format_stamp: "Formatted at 2026-09-28 21:12:40 on dist-test-slave-v7xz"
server_key: "0389349cd6c3e302b85f0aa35d9d9443"
server_key_iv: "3bac14a336203bff08ba9f10349e732d"
server_key_version: "encryptionkey@0"
I20260928 21:12:41.819608 9142 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:41.833537 9142 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:41.836802 9142 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:41.837065 9142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:41.837734 9142 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:41.838605 9142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:41.838652 9142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.838722 9142 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:41.838753 9142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:41.848057 9142 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40561
I20260928 21:12:41.848102 9290 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40561 every 8 connection(s)
I20260928 21:12:41.849179 9142 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:41.852375 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9142
I20260928 21:12:41.853080 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42733
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38079
--webserver_port=38423
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:41.865132 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.851375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:44747)
0928 21:12:41.851922 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:41.851976 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:41.854005 (+ 2029us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:41.855419 (+ 1414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:41.855426 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:41.855838 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:41.856531 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.856555 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.857943 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.857949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.859581 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.859598 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.860453 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.860461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.860711 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.861439 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:41.861467 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:41.864235 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":158,"threads_started":1}
W20260928 21:12:41.865516 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:41.851269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44747 (local address 127.0.98.62:38079)
0928 21:12:41.851738 (+ 469us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:41.851756 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:41.852099 (+ 343us) server_negotiation.cc:407] Connection header received
0928 21:12:41.854467 (+ 2368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:41.854486 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:41.854954 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:41.855346 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:41.856736 (+ 1390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.857795 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.859737 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.860268 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.860404 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.864585 (+ 4181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44747: BlockingRecv error: recv got EOF from 127.0.98.1:44747 (error 108)
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":112,"threads_started":1}
W20260928 21:12:41.866088 9291 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38079 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38079: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:41.872087 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.866552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:38803)
0928 21:12:41.866700 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:41.866715 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:41.866797 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:41.867038 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:41.867042 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:41.867230 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:41.867346 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.867353 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.868129 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.868132 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.869069 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.869080 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.869515 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.869519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.869614 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.870091 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:41.870105 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:41.871985 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 21:12:41.872263 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:41.866670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38803 (local address 127.0.98.62:38079)
0928 21:12:41.866780 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:41.866783 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:41.866796 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:41.866888 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:41.866890 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:41.866932 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:41.867000 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:41.867494 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.868028 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.869177 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.869386 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.869480 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.872148 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38803: BlockingRecv error: recv got EOF from 127.0.98.1:38803 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:41.878057 9294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:41.872774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:58509)
0928 21:12:41.872932 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:41.872946 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:41.873020 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:41.873243 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:41.873246 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:41.873405 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:41.873491 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.873497 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.874163 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.874167 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.875021 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:41.875032 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.875433 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.875436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:41.875536 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.875984 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:41.875998 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:41.877906 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 21:12:41.878145 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:41.872896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58509 (local address 127.0.98.62:38079)
0928 21:12:41.873002 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:41.873005 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:41.873016 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:12:41.873107 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:41.873109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:41.873145 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:41.873225 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:41.873599 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.874066 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.875133 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:41.875336 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:41.875408 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:41.878026 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58509: BlockingRecv error: recv got EOF from 127.0.98.1:58509 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:12:41.878268 9291 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:12:41.973071 9295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:41.973357 9295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:41.973402 9295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:41.977399 9295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:41.977464 9295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:41.977528 9295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:41.982810 9295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42733
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38423
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9295
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:41.983841 9295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:41.984812 9295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:41.994187 9303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.994194 9301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:41.994278 9300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:41.994397 9295 server_base.cc:1034] running on GCE node
I20260928 21:12:41.994894 9295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:41.995589 9295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:41.996795 9295 hybrid_clock.cc:648] HybridClock initialized: now 1790629961996765 us; error 40 us; skew 500 ppm
Sep 28 21:12:41 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629961, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:41.999811 9295 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260928 21:12:42.001025 9295 webserver.cc:460] Webserver started at http://127.0.98.2:38423/ using document root <none> and password file <none>
I20260928 21:12:42.001617 9295 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:42.001689 9295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:42.005674 9295 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260928 21:12:42.007818 9310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:42.009164 9295 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:42.009286 9295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "6cac52a2798c47bd8b54dfc13ea5a119"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "e76e6151a39a02dfbeeb383a6b16e7f2"
server_key_iv: "c64ef7d11796925898721ec56bcf0af9"
server_key_version: "encryptionkey@0"
I20260928 21:12:42.009835 9295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:42.035310 9295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:42.039161 9295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:42.039399 9295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:42.040084 9295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:42.041158 9295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:42.041234 9295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:42.041301 9295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:42.041337 9295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:42.051692 9295 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42733
I20260928 21:12:42.051715 9423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42733 every 8 connection(s)
I20260928 21:12:42.052995 9295 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:42.060765 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9295
I20260928 21:12:42.061331 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45005
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38079
--webserver_port=33659
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:12:42.065502 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.055148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52963 (local address 127.0.98.62:38079)
0928 21:12:42.055329 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.055335 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.056316 (+ 981us) server_negotiation.cc:407] Connection header received
0928 21:12:42.057739 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.057742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.057796 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.057879 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.059129 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.059654 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.061183 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.061438 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.061567 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.065311 (+ 3744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52963: BlockingRecv error: recv got EOF from 127.0.98.2:52963 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 21:12:42.065982 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.055632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:52963)
0928 21:12:42.056146 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.056193 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.057513 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.058117 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.058124 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.058486 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.058974 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.058988 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.059803 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.059806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.061006 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.061018 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.061655 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.061662 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.061848 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.062572 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.062597 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.065093 (+ 2496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":114,"threads_started":1}
W20260928 21:12:42.066638 9424 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38079 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38079: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:42.072675 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.067146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:55799)
0928 21:12:42.067326 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.067338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.067440 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.067711 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.067713 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.067886 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.067981 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.067987 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.068749 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.068754 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.069357 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.069365 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.069830 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.069834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.069924 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.070495 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.070507 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.072490 (+ 1983us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121}
W20260928 21:12:42.072737 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.067307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55799 (local address 127.0.98.62:38079)
0928 21:12:42.067444 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.067448 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.067465 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:12:42.067544 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.067547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.067591 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.067694 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.068102 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.068639 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.069474 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.069719 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.069831 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.072626 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55799: BlockingRecv error: recv got EOF from 127.0.98.2:55799 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:42.079264 9426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.073528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:60623)
0928 21:12:42.073658 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.073676 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.073745 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.074053 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.074055 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.074196 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.074274 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.074278 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.075291 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.075294 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.075821 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.075829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.076432 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.076436 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.076512 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.077092 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.077107 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.079118 (+ 2011us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 21:12:42.079492 9424 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:12:42.079525 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.073593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60623 (local address 127.0.98.62:38079)
0928 21:12:42.073707 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.073711 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.073726 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:42.073832 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.073835 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.073877 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.073965 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.074379 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.075166 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.075961 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.076334 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.076429 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.079396 (+ 2967us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60623: BlockingRecv error: recv got EOF from 127.0.98.2:60623 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260928 21:12:42.176512 9427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:42.176795 9427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:42.176905 9427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:42.181063 9427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:42.181128 9427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:42.181188 9427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:42.186378 9427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45005
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33659
--tserver_master_addrs=127.0.98.62:38079
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:42.187397 9427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:42.188560 9427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:42.198093 9432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:42.198143 9433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:42.198087 9435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:42.198171 9427 server_base.cc:1034] running on GCE node
I20260928 21:12:42.198858 9427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:42.199502 9427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:42.200682 9427 hybrid_clock.cc:648] HybridClock initialized: now 1790629962200660 us; error 36 us; skew 500 ppm
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629962, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:12:42.203869 9427 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260928 21:12:42.204684 9427 webserver.cc:460] Webserver started at http://127.0.98.3:33659/ using document root <none> and password file <none>
I20260928 21:12:42.205291 9427 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:42.205417 9427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:42.209147 9427 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 21:12:42.211220 9442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:42.212404 9427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:42.212543 9427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "039c8e74b286470c810f7742197eb562"
format_stamp: "Formatted at 2026-09-28 21:12:41 on dist-test-slave-v7xz"
server_key: "4ce648d3ed8501e5a151c5d550dc9466"
server_key_iv: "bbb0ce9e35c539a4d7da932c54a808db"
server_key_version: "encryptionkey@0"
I20260928 21:12:42.213177 9427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:42.240458 9427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:42.244110 9427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:42.244330 9427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:42.245219 9427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:42.246084 9427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:42.246155 9427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:42.246220 9427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:42.246249 9427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:42.255049 9427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45005
I20260928 21:12:42.255093 9555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45005 every 8 connection(s)
I20260928 21:12:42.256070 9427 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:42.258044 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9427
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260928 21:12:42.269392 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.258093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35293 (local address 127.0.98.62:38079)
0928 21:12:42.258239 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.258244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.259750 (+ 1506us) server_negotiation.cc:407] Connection header received
0928 21:12:42.261436 (+ 1686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.261439 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.261546 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.261666 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.263104 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.263798 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.265322 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.265543 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.265658 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.269315 (+ 3657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35293: BlockingRecv error: recv got EOF from 127.0.98.3:35293 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:12:42.269394 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.259211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57126 (local address 127.0.98.62:38079)
0928 21:12:42.259464 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.259469 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.259484 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:42.259537 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.259541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.259590 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.259696 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.260495 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.261420 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.262497 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.262834 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.262948 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.264633 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:42.264640 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:42.264658 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:12:42.264691 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:42.266936 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:42.267483 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:42.267488 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:42.267493 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:42.267664 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:12:42.267972 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:12:42.267976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:12:42.267978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:12:42.268388 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:42.268598 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:42.268923 (+ 325us) server_negotiation.cc:299] Negotiation successful
0928 21:12:42.269204 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":86,"threads_started":1}
I20260928 21:12:42.270217 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.258554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:35293)
0928 21:12:42.259584 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.259648 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.261169 (+ 1521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.261843 (+ 674us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.261851 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.262263 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.262942 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.262963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.263951 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.263954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.265176 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.265186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.265703 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.265707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.265928 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.266441 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.266460 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.269205 (+ 2745us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":159,"threads_started":1}
W20260928 21:12:42.270766 9556 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38079 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38079: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:42.276422 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.271221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:43563)
0928 21:12:42.271383 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.271399 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.271487 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.271805 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.271808 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.271949 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.272038 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.272043 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.272771 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.272776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.273404 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.273413 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.273853 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.273856 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.273948 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.274454 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.274474 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.276271 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100}
W20260928 21:12:42.276495 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.271319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43563 (local address 127.0.98.62:38079)
0928 21:12:42.271438 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.271442 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.271457 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:12:42.271578 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.271581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.271624 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.271695 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.272154 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.272656 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.273525 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.273742 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.273852 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.276384 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43563: BlockingRecv error: recv got EOF from 127.0.98.3:43563 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:42.281944 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.277000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:38599)
0928 21:12:42.277186 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.277202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.277285 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.277523 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.277526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.277692 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.277760 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.277765 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.278486 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.278491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.279026 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.279033 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.279493 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.279496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.279579 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.280097 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.280109 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.281794 (+ 1685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260928 21:12:42.282006 9293 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.277108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38599 (local address 127.0.98.62:38079)
0928 21:12:42.277266 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.277270 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.277283 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:42.277377 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.277380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.277417 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.277510 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.277868 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.278378 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.279159 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.279383 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.279494 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.281900 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38599: BlockingRecv error: recv got EOF from 127.0.98.3:38599 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
W20260928 21:12:42.282272 9556 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:42 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:42.884732 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:42.879008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:42721)
0928 21:12:42.879297 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:42.879310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:42.879404 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:42.879656 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:42.879659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:42.879831 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:42.879941 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.879946 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.880776 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.880779 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.881483 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:42.881493 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.881956 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.881959 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:42.882079 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.882622 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:42.882634 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:42.884565 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":85,"threads_started":1}
W20260928 21:12:42.884887 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:42.879088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42721 (local address 127.0.98.62:38079)
0928 21:12:42.879340 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:42.879344 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:42.879360 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:12:42.879494 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:42.879496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:42.879538 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:42.879616 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:42.880099 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.880646 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.881623 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:42.881847 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:42.881931 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:42.884709 (+ 2778us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42721: BlockingRecv error: recv got EOF from 127.0.98.1:42721 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":60,"threads_started":1}
Sep 28 21:12:43 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:43 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:43.086150 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:43.080173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:34609)
0928 21:12:43.080559 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:43.080572 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:43.080727 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:43.081026 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:43.081029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:43.081239 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:43.081350 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:43.081355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.082105 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.082111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:43.082683 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:43.082691 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.083182 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.083185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:43.083284 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:43.083830 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:43.083846 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:43.085946 (+ 2100us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":89,"threads_started":1}
W20260928 21:12:43.086179 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:43.080316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34609 (local address 127.0.98.62:38079)
0928 21:12:43.080485 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:43.080490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:43.080661 (+ 171us) server_negotiation.cc:407] Connection header received
0928 21:12:43.080861 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:43.080864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:43.080908 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:43.081004 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:43.081468 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.081984 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.082803 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.083061 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.083177 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:43.086087 (+ 2910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34609: BlockingRecv error: recv got EOF from 127.0.98.2:34609 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Sep 28 21:12:43 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:43 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:43.288813 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:43.283021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:49441)
0928 21:12:43.283296 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:43.283317 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:43.283430 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:43.283715 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:43.283718 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:43.283909 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:43.284054 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:43.284061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.285050 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.285054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:43.285683 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:43.285692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.286133 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.286136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:43.286215 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:43.286692 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:43.286702 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:43.288627 (+ 1925us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
W20260928 21:12:43.288895 9562 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:43.283112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49441 (local address 127.0.98.62:38079)
0928 21:12:43.283265 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:43.283271 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:43.283409 (+ 138us) server_negotiation.cc:407] Connection header received
0928 21:12:43.283509 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:43.283513 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:43.283575 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:43.283657 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:43.284222 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.284902 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.285799 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.286024 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.286107 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:43.288759 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:49441: BlockingRecv error: recv got EOF from 127.0.98.3:49441 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 28 21:12:43 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:43 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:43.890887 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:43.885701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:40603)
0928 21:12:43.885953 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:43.885965 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:43.886073 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:43.886305 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:43.886307 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:43.886474 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:43.886563 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:43.886568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.887320 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.887322 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:43.887885 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:43.887893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.888306 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.888308 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:43.888372 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:43.888951 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:43.888962 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:43.890750 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":83,"threads_started":1}
W20260928 21:12:43.891096 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:43.885762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40603 (local address 127.0.98.62:38079)
0928 21:12:43.886063 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:43.886067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:43.886081 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:12:43.886157 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:43.886159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:43.886199 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:43.886274 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:43.886695 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.887223 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.887987 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:43.888209 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:43.888286 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:43.890895 (+ 2609us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40603: BlockingRecv error: recv got EOF from 127.0.98.1:40603 (error 108)
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":89,"threads_started":1}
Sep 28 21:12:44 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:44 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:44.092523 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:44.086996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:59685)
0928 21:12:44.087268 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:44.087285 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:44.087431 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:44.087706 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:44.087709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:44.087909 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:44.088000 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:44.088006 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.088760 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.088766 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:44.089353 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:44.089361 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.089811 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.089814 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:44.089900 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:44.090424 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:44.090435 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:44.092325 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":84,"threads_started":1}
W20260928 21:12:44.092751 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:44.087058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59685 (local address 127.0.98.62:38079)
0928 21:12:44.087259 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:44.087264 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:44.087370 (+ 106us) server_negotiation.cc:407] Connection header received
0928 21:12:44.087528 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:44.087533 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:44.087583 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:44.087668 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:44.088118 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.088651 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.089472 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.089703 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.089809 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:44.092518 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59685: BlockingRecv error: recv got EOF from 127.0.98.2:59685 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
Sep 28 21:12:44 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:44 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:44.295260 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:44.289619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:32829)
0928 21:12:44.289893 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:44.289907 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:44.289994 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:44.290236 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:44.290238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:44.290413 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:44.290506 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:44.290512 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.291240 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.291242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:44.291802 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:44.291809 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.292401 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.292405 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:44.292471 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:44.293083 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:44.293098 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:44.295065 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":102,"threads_started":1}
W20260928 21:12:44.295286 9566 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:44.289825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32829 (local address 127.0.98.62:38079)
0928 21:12:44.289955 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:44.289958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:44.289971 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:12:44.290080 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:44.290082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:44.290124 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:44.290202 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:44.290621 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.291148 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.291907 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.292116 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.292256 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:44.295186 (+ 2930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:32829: BlockingRecv error: recv got EOF from 127.0.98.3:32829 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 28 21:12:44 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:44 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:44.902844 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:44.891591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:47067)
0928 21:12:44.891848 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:44.891861 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:44.891946 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:44.892187 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:44.892189 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:44.892363 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:44.892457 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:44.892462 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.893318 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.893321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:44.893882 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:44.893890 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.894290 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.894292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:44.894353 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:44.900743 (+ 6390us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:44.900756 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:44.902655 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
W20260928 21:12:44.902928 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:44.891675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47067 (local address 127.0.98.62:38079)
0928 21:12:44.891908 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:44.891912 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:44.891924 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:12:44.892026 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:44.892029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:44.892070 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:44.892146 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:44.892548 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.893223 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.893982 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:44.894195 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:44.894276 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:44.902801 (+ 8525us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47067: BlockingRecv error: recv got EOF from 127.0.98.1:47067 (error 108)
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
Sep 28 21:12:45 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:45 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:45.098964 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:45.093368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:53201)
0928 21:12:45.093644 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:45.093689 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:45.093792 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:45.094021 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:45.094023 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:45.094214 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:45.094313 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:45.094319 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.095271 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.095274 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:45.095840 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:45.095848 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.096298 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.096300 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:45.096364 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:45.096900 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:45.096911 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:45.098828 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":87,"threads_started":1}
W20260928 21:12:45.099102 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:45.093432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53201 (local address 127.0.98.62:38079)
0928 21:12:45.093572 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:45.093576 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:45.093758 (+ 182us) server_negotiation.cc:407] Connection header received
0928 21:12:45.093863 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:45.093866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:45.093908 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:45.093979 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:45.094528 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.095127 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.095950 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.096190 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.096275 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:45.098968 (+ 2693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53201: BlockingRecv error: recv got EOF from 127.0.98.2:53201 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 21:12:45 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:45 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:45.301486 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:45.296144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:38617)
0928 21:12:45.296411 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:45.296425 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:45.296525 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:45.296776 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:45.296778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:45.296970 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:45.297063 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:45.297068 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.297802 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.297804 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:45.298341 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:45.298348 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.298766 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.298768 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:45.298826 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:45.299294 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:45.299305 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:45.301339 (+ 2034us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":68,"threads_started":1}
W20260928 21:12:45.301609 9570 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:45.296295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:38617 (local address 127.0.98.62:38079)
0928 21:12:45.296445 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:45.296449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:45.296477 (+ 28us) server_negotiation.cc:407] Connection header received
0928 21:12:45.296612 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:45.296614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:45.296660 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:45.296727 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:45.297190 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.297707 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.298441 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.298662 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.298741 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:45.301469 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:38617: BlockingRecv error: recv got EOF from 127.0.98.3:38617 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 21:12:45 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:45 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:45.910444 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:45.903798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:40805)
0928 21:12:45.904190 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:45.904208 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:45.904337 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:45.904656 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:45.904659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:45.904913 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:45.905097 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:45.905107 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.906096 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.906099 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:45.906645 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:45.906652 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.907148 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.907151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:45.907215 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:45.907770 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:45.907782 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:45.910229 (+ 2447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":99,"threads_started":1}
W20260928 21:12:45.910549 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:45.903876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40805 (local address 127.0.98.62:38079)
0928 21:12:45.904179 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:45.904185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:45.904293 (+ 108us) server_negotiation.cc:407] Connection header received
0928 21:12:45.904442 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:45.904447 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:45.904502 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:45.904605 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:45.905249 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.905940 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.906756 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:45.906994 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:45.907116 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:45.910407 (+ 3291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40805: BlockingRecv error: recv got EOF from 127.0.98.1:40805 (error 108)
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":102,"threads_started":1}
Sep 28 21:12:46 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:46 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:46.105652 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:46.099828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:46843)
0928 21:12:46.100147 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:46.100162 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:46.100311 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:46.100590 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:46.100593 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:46.100869 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:46.100982 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:46.100988 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.101824 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.101829 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:46.102514 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:46.102527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.102982 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.102985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:46.103086 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:46.103669 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:46.103681 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:46.105514 (+ 1833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":112,"threads_started":1}
W20260928 21:12:46.105806 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:46.099968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46843 (local address 127.0.98.62:38079)
0928 21:12:46.100124 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:46.100129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:46.100242 (+ 113us) server_negotiation.cc:407] Connection header received
0928 21:12:46.100409 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:46.100412 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:46.100481 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:46.100571 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:46.101112 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.101707 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.102638 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.102866 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.102978 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:46.105647 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46843: BlockingRecv error: recv got EOF from 127.0.98.2:46843 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 21:12:46 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:46 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:46.307621 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:46.302374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:55599)
0928 21:12:46.302645 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:46.302659 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:46.302772 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:46.302986 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:46.302988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:46.303158 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:46.303255 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:46.303259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.303960 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.303962 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:46.304503 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:46.304510 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.304965 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.304969 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:46.305038 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:46.305624 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:46.305636 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:46.307497 (+ 1861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
W20260928 21:12:46.307770 9574 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:46.302418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55599 (local address 127.0.98.62:38079)
0928 21:12:46.302557 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:46.302562 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:46.302713 (+ 151us) server_negotiation.cc:407] Connection header received
0928 21:12:46.302839 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:46.302841 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:46.302886 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:46.302952 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:46.303351 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.303872 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.304598 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.304814 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.304953 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:46.307649 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55599: BlockingRecv error: recv got EOF from 127.0.98.3:55599 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 21:12:46 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:46 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:46.917550 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:46.911439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:47933)
0928 21:12:46.911807 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:46.911824 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:46.911944 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:46.912274 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:46.912277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:46.912517 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:46.912680 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:46.912692 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.913534 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.913536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:46.914231 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:46.914241 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.914664 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.914667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:46.914731 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:46.915336 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:46.915357 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:46.917366 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":84,"threads_started":1}
W20260928 21:12:46.917695 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:46.911498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47933 (local address 127.0.98.62:38079)
0928 21:12:46.911797 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:46.911802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:46.911906 (+ 104us) server_negotiation.cc:407] Connection header received
0928 21:12:46.912046 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:46.912049 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:46.912107 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:46.912229 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:46.912806 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.913433 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.914333 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:46.914552 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:46.914641 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:46.917493 (+ 2852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47933: BlockingRecv error: recv got EOF from 127.0.98.1:47933 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":81,"threads_started":1}
Sep 28 21:12:47 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:47 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:47.112478 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:47.106460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:49981)
0928 21:12:47.106769 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:47.106783 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:47.106879 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:47.107242 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:47.107244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:47.107426 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:47.107524 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:47.107530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.108341 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.108343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:47.109028 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:47.109038 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.109527 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.109530 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:47.109596 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:47.110187 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:47.110201 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:47.112289 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":134,"threads_started":1}
W20260928 21:12:47.112524 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:47.106611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49981 (local address 127.0.98.62:38079)
0928 21:12:47.106751 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:47.106755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:47.106838 (+ 83us) server_negotiation.cc:407] Connection header received
0928 21:12:47.106961 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:47.106963 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:47.107008 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:47.107083 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:47.107676 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.108218 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.109186 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.109411 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.109515 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:47.112416 (+ 2901us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:49981: BlockingRecv error: recv got EOF from 127.0.98.2:49981 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 28 21:12:47 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:47 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:47.314499 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:47.308520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:42203)
0928 21:12:47.308791 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:47.308811 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:47.308951 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:47.309250 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:47.309252 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:47.309519 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:47.309646 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:47.309658 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.310649 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.310652 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:47.311288 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:47.311299 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.311788 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.311790 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:47.311849 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:47.312323 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:47.312335 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:47.314357 (+ 2022us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":104,"threads_started":1}
W20260928 21:12:47.314538 9578 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:47.308607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42203 (local address 127.0.98.62:38079)
0928 21:12:47.308771 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:47.308775 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:47.308917 (+ 142us) server_negotiation.cc:407] Connection header received
0928 21:12:47.309058 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:47.309060 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:47.309104 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:47.309220 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:47.309773 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.310535 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.311455 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.311692 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.311771 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:47.314446 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42203: BlockingRecv error: recv got EOF from 127.0.98.3:42203 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Sep 28 21:12:47 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:47 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:47.923611 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:47.918398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:40537)
0928 21:12:47.918659 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:47.918672 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:47.918769 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:47.919012 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:47.919014 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:47.919188 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:47.919297 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:47.919303 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.920043 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.920046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:47.920593 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:47.920602 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.921042 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.921044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:47.921105 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:47.921580 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:47.921591 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:47.923490 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":88,"threads_started":1}
W20260928 21:12:47.923770 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:47.918451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40537 (local address 127.0.98.62:38079)
0928 21:12:47.918680 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:47.918684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:47.918741 (+ 57us) server_negotiation.cc:407] Connection header received
0928 21:12:47.918851 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:47.918853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:47.918895 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:47.918974 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:47.919407 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.919952 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.920707 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:47.920949 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:47.921027 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:47.923626 (+ 2599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40537: BlockingRecv error: recv got EOF from 127.0.98.1:40537 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
Sep 28 21:12:48 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:48 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:48.119788 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:48.113484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:57413)
0928 21:12:48.113812 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:48.113828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:48.113989 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:48.114361 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:48.114365 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:48.114576 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:48.114675 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:48.114681 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.115709 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.115715 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:48.116298 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:48.116307 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.116812 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.116815 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:48.116932 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:48.117561 (+ 629us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:48.117574 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:48.119661 (+ 2087us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":102,"threads_started":1}
W20260928 21:12:48.120136 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:48.113589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57413 (local address 127.0.98.62:38079)
0928 21:12:48.113737 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:48.113743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:48.114010 (+ 267us) server_negotiation.cc:407] Connection header received
0928 21:12:48.114078 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:48.114081 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:48.114152 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:48.114284 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:48.114818 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.115592 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.116426 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.116673 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.116813 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:48.119919 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57413: BlockingRecv error: recv got EOF from 127.0.98.2:57413 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:12:48 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:48 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:48.320787 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:48.315375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:53923)
0928 21:12:48.315706 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:48.315718 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:48.315865 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:48.316142 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:48.316144 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:48.316359 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:48.316453 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:48.316459 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.317304 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.317309 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:48.317885 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:48.317893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.318365 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.318368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:48.318455 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:48.318934 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:48.318945 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:48.320679 (+ 1734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":98,"threads_started":1}
W20260928 21:12:48.320992 9582 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:48.315422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53923 (local address 127.0.98.62:38079)
0928 21:12:48.315571 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:48.315576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:48.315803 (+ 227us) server_negotiation.cc:407] Connection header received
0928 21:12:48.315970 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:48.315973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:48.316029 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:48.316115 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:48.316571 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.317191 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.318008 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.318245 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.318365 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:48.320811 (+ 2446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53923: BlockingRecv error: recv got EOF from 127.0.98.3:53923 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:12:48 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:48 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:48.930460 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:48.924546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:44071)
0928 21:12:48.924905 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:48.924919 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:48.925025 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:48.925383 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:48.925385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:48.925582 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:48.925698 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:48.925704 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.926535 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.926538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:48.927098 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:48.927107 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.927569 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.927573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:48.927652 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:48.928325 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:48.928338 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:48.930287 (+ 1949us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":99,"threads_started":1}
W20260928 21:12:48.930536 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:48.924638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44071 (local address 127.0.98.62:38079)
0928 21:12:48.924915 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:48.924919 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:48.924995 (+ 76us) server_negotiation.cc:407] Connection header received
0928 21:12:48.925156 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:48.925159 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:48.925216 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:48.925316 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:48.925834 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.926421 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.927212 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:48.927431 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:48.927512 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:48.930412 (+ 2900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44071: BlockingRecv error: recv got EOF from 127.0.98.1:44071 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":108,"threads_started":1}
Sep 28 21:12:49 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:49 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:49.126289 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:49.120538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:59359)
0928 21:12:49.120871 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:49.120884 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:49.120982 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:49.121199 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:49.121201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:49.121375 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:49.121472 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:49.121477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.122201 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.122204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:49.122907 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:49.122917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.123462 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.123466 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:49.123536 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:49.124014 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:49.124026 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:49.126092 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
W20260928 21:12:49.126313 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:49.120591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59359 (local address 127.0.98.62:38079)
0928 21:12:49.120748 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:49.120752 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:49.120952 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:12:49.121048 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:49.121050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:49.121092 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:49.121161 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:49.121566 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.122093 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.123018 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.123296 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.123379 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:49.126214 (+ 2835us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59359: BlockingRecv error: recv got EOF from 127.0.98.2:59359 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 28 21:12:49 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:49 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:49.326973 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:49.321628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:58835)
0928 21:12:49.321953 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:49.321966 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:49.322066 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:49.322322 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:49.322324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:49.322500 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:49.322643 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:49.322650 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.323370 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.323372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:49.324002 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:49.324011 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.324445 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.324448 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:49.324508 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:49.324998 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:49.325009 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:49.326854 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":114,"threads_started":1}
W20260928 21:12:49.327188 9586 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:49.321699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58835 (local address 127.0.98.62:38079)
0928 21:12:49.321817 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:49.321820 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:49.322019 (+ 199us) server_negotiation.cc:407] Connection header received
0928 21:12:49.322162 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:49.322164 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:49.322208 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:49.322280 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:49.322755 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.323278 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.324135 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.324347 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.324428 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:49.326985 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58835: BlockingRecv error: recv got EOF from 127.0.98.3:58835 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 28 21:12:49 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:49 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:49.940676 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:49.931353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:33811)
0928 21:12:49.931626 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:49.931650 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:49.931746 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:49.932105 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:49.932108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:49.932342 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:49.932472 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:49.932478 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.933680 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.933683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:49.934252 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:49.934261 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.934720 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.934722 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:49.934783 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:49.938744 (+ 3961us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:49.938757 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:49.940555 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":102,"threads_started":1}
W20260928 21:12:49.940959 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:49.931584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33811 (local address 127.0.98.62:38079)
0928 21:12:49.931828 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:49.931831 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:49.931873 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:12:49.931914 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:49.931916 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:49.931957 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:49.932064 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:49.932776 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.933545 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.934383 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:49.934613 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:49.934702 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:49.940789 (+ 6087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33811: BlockingRecv error: recv got EOF from 127.0.98.1:33811 (error 108)
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":87,"threads_started":1}
Sep 28 21:12:50 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:50 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:50.133877 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:50.127250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:35587)
0928 21:12:50.127620 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:50.127639 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:50.127803 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:50.128107 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:50.128110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:50.128331 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:50.128427 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:50.128433 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.129548 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.129554 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:50.130223 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:50.130232 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.130989 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.130993 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:50.131062 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:50.131692 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:50.131705 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:50.133691 (+ 1986us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":87,"threads_started":1}
W20260928 21:12:50.133942 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:50.127373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35587 (local address 127.0.98.62:38079)
0928 21:12:50.127588 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:50.127602 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:50.127738 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:12:50.127921 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:50.127925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:50.127983 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:50.128079 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:50.128555 (+ 476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.129425 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.130360 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.130694 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.130831 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:50.133824 (+ 2993us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35587: BlockingRecv error: recv got EOF from 127.0.98.2:35587 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
Sep 28 21:12:50 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:50 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:50.333446 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:50.327838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:53235)
0928 21:12:50.328098 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:50.328111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:50.328207 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:50.328456 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:50.328458 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:50.328643 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:50.328736 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:50.328741 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.329537 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.329539 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:50.330113 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:50.330120 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.330530 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.330532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:50.330591 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:50.331074 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:50.331085 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:50.333274 (+ 2189us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":67,"threads_started":1}
W20260928 21:12:50.333496 9590 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:50.327979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53235 (local address 127.0.98.62:38079)
0928 21:12:50.328148 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:50.328153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:50.328177 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:12:50.328298 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:50.328300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:50.328345 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:50.328418 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:50.328878 (+ 460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.329420 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.330219 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.330432 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.330509 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:50.333394 (+ 2885us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53235: BlockingRecv error: recv got EOF from 127.0.98.3:53235 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 28 21:12:50 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:50 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:50.947708 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:50.941692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:48991)
0928 21:12:50.942015 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:50.942030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:50.942135 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:50.942402 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:50.942405 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:50.942670 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:50.942799 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:50.942806 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.943840 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.943843 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:50.944478 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:50.944488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.945017 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.945021 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:50.945084 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:50.945693 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:50.945714 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:50.947581 (+ 1867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":100,"threads_started":1}
W20260928 21:12:50.947890 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:50.941769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48991 (local address 127.0.98.62:38079)
0928 21:12:50.942070 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:50.942074 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:50.942090 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:12:50.942212 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:50.942215 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:50.942257 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:50.942355 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:50.942925 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.943711 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.944638 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:50.944895 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:50.944987 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:50.947743 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48991: BlockingRecv error: recv got EOF from 127.0.98.1:48991 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":118,"threads_started":1}
Sep 28 21:12:51 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:51 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:51.140645 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:51.134780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:47343)
0928 21:12:51.135110 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:51.135125 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:51.135265 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:51.135566 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:51.135570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:51.135789 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:51.135884 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:51.135890 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.136686 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.136692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:51.137361 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:51.137370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.137842 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.137846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:51.137922 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:51.138467 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:51.138479 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:51.140524 (+ 2045us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":91,"threads_started":1}
W20260928 21:12:51.140807 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:51.134900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47343 (local address 127.0.98.62:38079)
0928 21:12:51.135068 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:51.135073 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:51.135210 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:12:51.135397 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:51.135400 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:51.135454 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:51.135546 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:51.136011 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.136560 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.137487 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.137723 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.137846 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:51.140676 (+ 2830us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47343: BlockingRecv error: recv got EOF from 127.0.98.2:47343 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Sep 28 21:12:51 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:51 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:51.339771 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:51.334223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:55405)
0928 21:12:51.334553 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:51.334568 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:51.334683 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:51.334933 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:51.334935 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:51.335150 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:51.335242 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:51.335247 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.335974 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.335977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:51.336573 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:51.336581 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.337049 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.337051 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:51.337138 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:51.337727 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:51.337738 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:51.339652 (+ 1914us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
W20260928 21:12:51.340008 9594 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:51.334303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55405 (local address 127.0.98.62:38079)
0928 21:12:51.334450 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:51.334455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:51.334651 (+ 196us) server_negotiation.cc:407] Connection header received
0928 21:12:51.334763 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:51.334765 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:51.334811 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:51.334894 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:51.335343 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.335868 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.336696 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.336950 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.337031 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:51.339846 (+ 2815us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55405: BlockingRecv error: recv got EOF from 127.0.98.3:55405 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 28 21:12:51 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:51 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:51.954010 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:51.948612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:45541)
0928 21:12:51.948939 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:51.948953 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:51.949047 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:51.949288 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:51.949290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:51.949481 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:51.949576 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:51.949581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.950301 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.950304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:51.950895 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:51.950903 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.951318 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.951321 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:51.951392 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:51.951979 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:51.951990 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:51.953894 (+ 1904us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":84,"threads_started":1}
W20260928 21:12:51.954211 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:51.948669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45541 (local address 127.0.98.62:38079)
0928 21:12:51.948941 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:51.948945 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:51.949029 (+ 84us) server_negotiation.cc:407] Connection header received
0928 21:12:51.949131 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:51.949133 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:51.949176 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:51.949251 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:51.949677 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.950206 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.951001 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:51.951220 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:51.951295 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:51.954040 (+ 2745us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45541: BlockingRecv error: recv got EOF from 127.0.98.1:45541 (error 108)
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":75,"threads_started":1}
Sep 28 21:12:52 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:52 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:52.147452 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:52.141595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:50817)
0928 21:12:52.141886 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:52.141908 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:52.142044 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:52.142318 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:52.142320 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:52.142594 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:52.142689 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:52.142695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.143467 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.143473 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:52.144037 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:52.144044 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.144511 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.144514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:52.144597 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:52.145134 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:52.145146 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:52.147283 (+ 2137us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":82,"threads_started":1}
W20260928 21:12:52.147511 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:52.141682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50817 (local address 127.0.98.62:38079)
0928 21:12:52.141846 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:52.141851 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:52.141984 (+ 133us) server_negotiation.cc:407] Connection header received
0928 21:12:52.142145 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:52.142148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:52.142197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:52.142298 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:52.142821 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.143352 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.144157 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.144391 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.144509 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:52.147405 (+ 2896us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50817: BlockingRecv error: recv got EOF from 127.0.98.2:50817 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 28 21:12:52 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:52 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:52.346490 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:52.340665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:56195)
0928 21:12:52.340956 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:52.340978 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:52.341106 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:52.341365 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:52.341367 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:52.341544 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:52.341645 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:52.341651 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.342547 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.342550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:52.343163 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:52.343172 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.343662 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.343665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:52.343754 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:52.344276 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:52.344287 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:52.346319 (+ 2032us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":104,"threads_started":1}
W20260928 21:12:52.346669 9598 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:52.340807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56195 (local address 127.0.98.62:38079)
0928 21:12:52.340964 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:52.340971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:52.341041 (+ 70us) server_negotiation.cc:407] Connection header received
0928 21:12:52.341210 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:52.341212 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:52.341257 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:52.341348 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:52.341741 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.342412 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.343259 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.343486 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.343565 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:52.346512 (+ 2947us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56195: BlockingRecv error: recv got EOF from 127.0.98.3:56195 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Sep 28 21:12:52 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:52 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:52.960806 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:52.954884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:35597)
0928 21:12:52.955174 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:52.955188 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:52.955315 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:52.955652 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:52.955654 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:52.955897 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:52.956042 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:52.956051 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.956925 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.956928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:52.957596 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:52.957606 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.958042 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.958044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:52.958105 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:52.958670 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:52.958691 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:52.960611 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":111,"threads_started":1}
W20260928 21:12:52.960889 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:52.954976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35597 (local address 127.0.98.62:38079)
0928 21:12:52.955228 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:52.955233 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:52.955254 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:12:52.955437 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:52.955440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:52.955497 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:52.955612 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:52.956184 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.956773 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.957718 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:52.957943 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:52.958026 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:52.960740 (+ 2714us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35597: BlockingRecv error: recv got EOF from 127.0.98.1:35597 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
Sep 28 21:12:53 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:53 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:53.153699 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:53.148338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:41043)
0928 21:12:53.148618 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:53.148642 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:53.148740 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:53.148970 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:53.148972 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:53.149190 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:53.149289 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:53.149294 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.150051 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.150054 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:53.150612 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:53.150621 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.151030 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.151032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:53.151091 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:53.151607 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:53.151636 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:53.153525 (+ 1889us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":117,"threads_started":1}
W20260928 21:12:53.153764 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:53.148434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41043 (local address 127.0.98.62:38079)
0928 21:12:53.148577 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:53.148581 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:53.148712 (+ 131us) server_negotiation.cc:407] Connection header received
0928 21:12:53.148813 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:53.148815 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:53.148882 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:53.148951 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:53.149405 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.149924 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.150718 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.150935 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.151008 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:53.153647 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41043: BlockingRecv error: recv got EOF from 127.0.98.2:41043 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 21:12:53 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:53 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:53.352974 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:53.347262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:35409)
0928 21:12:53.347595 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:53.347635 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:53.347769 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:53.348049 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:53.348052 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:53.348262 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:53.348352 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:53.348357 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.349219 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.349223 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:53.349789 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:53.349797 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.350256 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.350260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:53.350342 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:53.350902 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:53.350914 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:53.352777 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":84,"threads_started":1}
W20260928 21:12:53.353020 9602 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:53.347391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35409 (local address 127.0.98.62:38079)
0928 21:12:53.347569 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:53.347574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:53.347718 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:12:53.347881 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:53.347885 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:53.347939 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:53.348030 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:53.348469 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.349058 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.349907 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.350136 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.350244 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:53.352919 (+ 2675us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35409: BlockingRecv error: recv got EOF from 127.0.98.3:35409 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Sep 28 21:12:53 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:53 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:53.967168 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:53.961782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:42547)
0928 21:12:53.962069 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:53.962083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:53.962178 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:53.962437 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:53.962439 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:53.962618 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:53.962711 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:53.962716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.963499 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.963501 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:53.964036 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:53.964043 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.964524 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.964526 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:53.964585 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:53.965220 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:53.965231 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:53.967051 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":99,"threads_started":1}
W20260928 21:12:53.967384 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:53.961866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42547 (local address 127.0.98.62:38079)
0928 21:12:53.962137 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:53.962141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:53.962194 (+ 53us) server_negotiation.cc:407] Connection header received
0928 21:12:53.962239 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:53.962241 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:53.962293 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:53.962382 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:53.962837 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.963404 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.964133 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:53.964409 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:53.964516 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:53.967247 (+ 2731us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42547: BlockingRecv error: recv got EOF from 127.0.98.1:42547 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
Sep 28 21:12:54 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:54 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:54.159799 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:54.154517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:32885)
0928 21:12:54.154789 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:54.154803 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:54.154906 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:54.155124 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:54.155126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:54.155301 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:54.155405 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:54.155410 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.156115 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.156117 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:54.156678 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:54.156685 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.157237 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.157240 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:54.157303 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:54.157878 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:54.157898 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:54.159681 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":103,"threads_started":1}
W20260928 21:12:54.160017 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:54.154588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32885 (local address 127.0.98.62:38079)
0928 21:12:54.154764 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:54.154768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:54.154880 (+ 112us) server_negotiation.cc:407] Connection header received
0928 21:12:54.154971 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:54.154973 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:54.155015 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:54.155088 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:54.155502 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.156024 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.156798 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.157094 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.157230 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:54.159878 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:32885: BlockingRecv error: recv got EOF from 127.0.98.2:32885 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 28 21:12:54 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:54 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:54.359211 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:54.353800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:55643)
0928 21:12:54.354044 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:54.354058 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:54.354153 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:54.354414 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:54.354416 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:54.354592 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:54.354688 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:54.354693 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.355405 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.355408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:54.355969 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:54.355977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.356398 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.356400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:54.356468 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:54.357048 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:54.357060 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:54.359031 (+ 1971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":81,"threads_started":1}
W20260928 21:12:54.359308 9606 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:54.353837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55643 (local address 127.0.98.62:38079)
0928 21:12:54.353960 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:54.353963 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:54.354124 (+ 161us) server_negotiation.cc:407] Connection header received
0928 21:12:54.354260 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:54.354262 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:54.354305 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:54.354384 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:54.354782 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.355286 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.356073 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.356281 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.356367 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:54.359186 (+ 2819us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55643: BlockingRecv error: recv got EOF from 127.0.98.3:55643 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Sep 28 21:12:54 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:54 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:54.977563 9609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:54.967926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:37829)
0928 21:12:54.968280 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:54.968295 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:54.968406 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:54.968649 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:54.968652 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:54.968892 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:54.969001 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:54.969008 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.969755 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.969758 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:54.970406 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:54.970415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.970841 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.970843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:54.970908 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:54.975110 (+ 4202us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:54.975124 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:54.977384 (+ 2260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":74,"threads_started":1}
W20260928 21:12:54.977751 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:54.968004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37829 (local address 127.0.98.62:38079)
0928 21:12:54.968286 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:54.968290 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:54.968385 (+ 95us) server_negotiation.cc:407] Connection header received
0928 21:12:54.968488 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:54.968490 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:54.968534 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:54.968611 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:54.969102 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.969641 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.970519 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:54.970739 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:54.970815 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:54.977521 (+ 6706us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37829: BlockingRecv error: recv got EOF from 127.0.98.1:37829 (error 108)
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":86,"threads_started":1}
Sep 28 21:12:55 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:55 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:55.166303 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:55.160539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:47539)
0928 21:12:55.161004 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:55.161017 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:55.161147 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:55.161484 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:55.161487 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:55.161689 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:55.161780 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:55.161786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.162577 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.162582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:55.163156 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:55.163164 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.163634 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.163638 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:55.163722 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:55.164262 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:55.164273 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:55.166118 (+ 1845us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":241,"threads_started":1}
W20260928 21:12:55.166383 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:55.160764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47539 (local address 127.0.98.62:38079)
0928 21:12:55.160943 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:55.160949 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:55.161092 (+ 143us) server_negotiation.cc:407] Connection header received
0928 21:12:55.161270 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:55.161273 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:55.161331 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:55.161441 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:55.161905 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.162460 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.163277 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.163515 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.163629 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:55.166257 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47539: BlockingRecv error: recv got EOF from 127.0.98.2:47539 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 28 21:12:55 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:55 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:55.365661 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:55.360054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:45387)
0928 21:12:55.360435 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:55.360448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:55.360586 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:55.360889 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:55.360892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:55.361102 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:55.361190 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:55.361196 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.361956 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.361962 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:55.362531 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:55.362539 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.363005 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.363009 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:55.363098 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:55.363624 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:55.363635 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:55.365490 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":156,"threads_started":1}
W20260928 21:12:55.365723 9610 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:55.360180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45387 (local address 127.0.98.62:38079)
0928 21:12:55.360339 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:55.360344 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:55.360528 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:12:55.360693 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:55.360697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:55.360752 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:55.360883 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:55.361309 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.361838 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.362656 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.362891 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.363003 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:55.365617 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45387: BlockingRecv error: recv got EOF from 127.0.98.3:45387 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 28 21:12:55 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:55 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:55.984640 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:55.978426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:49289)
0928 21:12:55.978758 (+ 332us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:55.978779 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:55.978890 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:55.979155 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:55.979157 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:55.979386 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:55.979486 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:55.979492 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.980270 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.980273 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:55.980930 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:55.980938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.981372 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.981375 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:55.981440 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:55.982120 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:55.982133 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:55.984501 (+ 2368us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":83,"threads_started":1}
W20260928 21:12:55.984804 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:55.978513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49289 (local address 127.0.98.62:38079)
0928 21:12:55.978871 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:55.978876 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:55.978894 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:12:55.978986 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:55.978988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:55.979031 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:55.979112 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:55.979599 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.980131 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.981050 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:55.981270 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:55.981346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:55.984649 (+ 3303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49289: BlockingRecv error: recv got EOF from 127.0.98.1:49289 (error 108)
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":92,"threads_started":1}
Sep 28 21:12:56 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:56 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:56.172909 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:56.167217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:34463)
0928 21:12:56.167503 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:56.167516 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:56.167614 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:56.167869 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:56.167871 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:56.168046 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:56.168139 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:56.168144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.168929 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.168932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:56.169484 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:56.169491 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.170103 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.170107 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:56.170204 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:56.170796 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:56.170829 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:56.172695 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":99,"threads_started":1}
W20260928 21:12:56.172986 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:56.167359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34463 (local address 127.0.98.62:38079)
0928 21:12:56.167500 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:56.167511 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:56.167590 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:12:56.167710 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:56.167712 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:56.167756 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:56.167830 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:56.168240 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.168770 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.169614 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.169944 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.170067 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:56.172871 (+ 2804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:34463: BlockingRecv error: recv got EOF from 127.0.98.2:34463 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 28 21:12:56 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:56 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:56.372192 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:56.366639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.3:53453)
0928 21:12:56.366965 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:56.366977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:56.367125 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:56.367440 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:56.367445 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:56.367634 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:56.367725 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:56.367731 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.368614 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.368617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:56.369256 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:56.369264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.369664 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.369667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:56.369740 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:56.370212 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:56.370222 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:56.372004 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":89,"threads_started":1}
W20260928 21:12:56.372222 9614 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:56.366743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53453 (local address 127.0.98.62:38079)
0928 21:12:56.366885 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:56.366890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:56.367105 (+ 215us) server_negotiation.cc:407] Connection header received
0928 21:12:56.367245 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:56.367248 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:56.367307 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:56.367392 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:56.367841 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.368517 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.369357 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.369567 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.369641 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:56.372121 (+ 2480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53453: BlockingRecv error: recv got EOF from 127.0.98.3:53453 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 28 21:12:56 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:56 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:56.991055 9617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:56.985596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.1:33935)
0928 21:12:56.985863 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:56.985878 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:56.986014 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:56.986378 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:56.986384 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:56.986629 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:56.986759 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:56.986766 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.987568 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.987570 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:56.988115 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:56.988122 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.988565 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.988567 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:56.988627 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:56.989206 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:56.989218 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:56.990930 (+ 1712us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":81,"threads_started":1}
W20260928 21:12:56.991242 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:56.985636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33935 (local address 127.0.98.62:38079)
0928 21:12:56.985916 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:56.985921 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:56.985960 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:12:56.986128 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:56.986131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:56.986241 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:56.986343 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:56.986891 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.987470 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.988244 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:56.988467 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:56.988544 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:56.991068 (+ 2524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33935: BlockingRecv error: recv got EOF from 127.0.98.1:33935 (error 108)
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":85,"threads_started":1}
Sep 28 21:12:57 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 28 21:12:57 dist-test-slave-v7xz krb5kdc[8559](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260928 21:12:57.180142 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:57.173951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38079 (local address 127.0.98.2:35139)
0928 21:12:57.174223 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:57.174244 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:57.174360 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:57.174633 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:57.174636 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:57.174848 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:57.174938 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:57.174943 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.175731 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.175735 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:57.176524 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:57.176534 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.177128 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.177131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:57.177223 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.177896 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:12:57.177908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:57.180013 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38079: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":83,"threads_started":1}
W20260928 21:12:57.180323 9618 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:57.174016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35139 (local address 127.0.98.62:38079)
0928 21:12:57.174175 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:57.174179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:57.174315 (+ 136us) server_negotiation.cc:407] Connection header received
0928 21:12:57.174466 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:57.174471 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:57.174516 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:57.174616 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:57.175056 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.175597 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.176680 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.177011 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.177124 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.180163 (+ 3039us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:35139: BlockingRecv error: recv got EOF from 127.0.98.2:35139 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:12:57.266290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9142
I20260928 21:12:57.272318 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9295
I20260928 21:12:57.277920 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9427
I20260928 21:12:57.283337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9072
2026-09-28T21:12:57Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18052 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-28T21:12:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:57Z Disabled control of system clock
I20260928 21:12:57.314052 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44901
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44901
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260928 21:12:57.424984 9623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:57.425375 9623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:57.425438 9623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:57.429693 9623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:57.429752 9623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:57.429768 9623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:57.429785 9623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:57.429829 9623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:57.434830 9623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44901
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:57.435936 9623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:57.437397 9623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:57.445987 9623 server_base.cc:1034] running on GCE node
W20260928 21:12:57.446091 9628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:57.446213 9631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:57.446069 9629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:57.446924 9623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:57.448016 9623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:57.449177 9623 hybrid_clock.cc:648] HybridClock initialized: now 1790629977449165 us; error 25 us; skew 500 ppm
I20260928 21:12:57.451016 9623 webserver.cc:460] Webserver started at http://127.0.98.62:38423/ using document root <none> and password file <none>
I20260928 21:12:57.451612 9623 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:57.451675 9623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:57.451917 9623 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:57.454023 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "3dafb3613478414cadb0f280de0a9ffa"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "1568f942abfff9d272165a940d6a783b"
server_key_iv: "07f04152301773ad7592064a2a7dd391"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.454588 9623 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "3dafb3613478414cadb0f280de0a9ffa"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "1568f942abfff9d272165a940d6a783b"
server_key_iv: "07f04152301773ad7592064a2a7dd391"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.458310 9623 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:12:57.460659 9637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.461877 9623 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:57.462028 9623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "3dafb3613478414cadb0f280de0a9ffa"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "1568f942abfff9d272165a940d6a783b"
server_key_iv: "07f04152301773ad7592064a2a7dd391"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.462139 9623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:57.476963 9623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:57.477622 9623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:57.477816 9623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:57.484536 9623 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44901
I20260928 21:12:57.484565 9689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44901 every 8 connection(s)
I20260928 21:12:57.485530 9623 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:57.488499 9690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:57.489812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9623
I20260928 21:12:57.489933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:57.490236 392 external_mini_cluster.cc:1428] Setting key 3f42d36881d5d3f8583c70be27405211
I20260928 21:12:57.494500 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa: Bootstrap starting.
I20260928 21:12:57.496657 9690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:57.497578 9690 log.cc:824] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:57.499418 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:57.491664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46358 (local address 127.0.98.62:44901)
0928 21:12:57.492305 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:57.492320 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:57.492354 (+ 34us) server_negotiation.cc:407] Connection header received
0928 21:12:57.493069 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:57.493087 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:57.493483 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:57.493822 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:57.494242 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.495264 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.496156 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.496630 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.496810 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.497749 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:57.497757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:57.497774 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:57.498153 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:57.498214 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:57.498234 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:57.498341 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:57.498492 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:12:57.498738 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":116,"threads_started":1}
I20260928 21:12:57.499552 9690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa: No bootstrap required, opened a new log
I20260928 21:12:57.503010 9690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafb3613478414cadb0f280de0a9ffa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260928 21:12:57.503242 9690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:57.503320 9690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3dafb3613478414cadb0f280de0a9ffa, State: Initialized, Role: FOLLOWER
I20260928 21:12:57.503767 9690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafb3613478414cadb0f280de0a9ffa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260928 21:12:57.503912 9690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:57.503996 9690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:57.504089 9690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:57.505059 9690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafb3613478414cadb0f280de0a9ffa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260928 21:12:57.505374 9690 leader_election.cc:302] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3dafb3613478414cadb0f280de0a9ffa; no voters:
I20260928 21:12:57.505656 9690 leader_election.cc:288] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:57.505772 9695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:57.506036 9695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [term 1 LEADER]: Becoming Leader. State: Replica: 3dafb3613478414cadb0f280de0a9ffa, State: Running, Role: LEADER
I20260928 21:12:57.506420 9695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafb3613478414cadb0f280de0a9ffa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } }
I20260928 21:12:57.506897 9690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:57.507951 9697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3dafb3613478414cadb0f280de0a9ffa. Latest consensus state: current_term: 1 leader_uuid: "3dafb3613478414cadb0f280de0a9ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafb3613478414cadb0f280de0a9ffa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } } }
I20260928 21:12:57.508069 9697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:57.508339 9696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3dafb3613478414cadb0f280de0a9ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3dafb3613478414cadb0f280de0a9ffa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44901 } } }
I20260928 21:12:57.508517 9696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:57.508455 9704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:57.511178 9704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:57.516319 9704 catalog_manager.cc:1347] Generated new cluster ID: 8c658bc9d1a54bee93b671dfd61ae6aa
I20260928 21:12:57.516392 9704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:57.542685 9704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:57.543998 9704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:57.563200 9704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3dafb3613478414cadb0f280de0a9ffa: Generated new TSK 0
I20260928 21:12:57.563969 9704 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:57.567857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--builtin_ntp_servers=127.0.98.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260928 21:12:57.679363 9714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:57.679622 9714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:57.679668 9714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:57.683627 9714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:57.683679 9714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:57.683732 9714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:57.688452 9714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:57.689491 9714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:57.690804 9714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:57.700273 9722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:57.700237 9719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:57.700374 9714 server_base.cc:1034] running on GCE node
W20260928 21:12:57.700307 9720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:57.700919 9714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:57.701588 9714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:57.702798 9714 hybrid_clock.cc:648] HybridClock initialized: now 1790629977702767 us; error 45 us; skew 500 ppm
I20260928 21:12:57.704684 9714 webserver.cc:460] Webserver started at http://127.0.98.1:38927/ using document root <none> and password file <none>
I20260928 21:12:57.705348 9714 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:57.705447 9714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:57.705687 9714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:57.707916 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "7fbc03238343471584abefc62927b1c8"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "67ef212807fef620ed4eb1534c6f1a55"
server_key_iv: "9c02d79e436eae0bd851bc97af646a0a"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.708527 9714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "7fbc03238343471584abefc62927b1c8"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "67ef212807fef620ed4eb1534c6f1a55"
server_key_iv: "9c02d79e436eae0bd851bc97af646a0a"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.712229 9714 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:57.714987 9728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.716260 9714 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:57.716405 9714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "7fbc03238343471584abefc62927b1c8"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "67ef212807fef620ed4eb1534c6f1a55"
server_key_iv: "9c02d79e436eae0bd851bc97af646a0a"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.716524 9714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:57.728220 9714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:57.728996 9714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:57.729185 9714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:57.729799 9714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:57.730722 9714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:57.730789 9714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.730854 9714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:57.730902 9714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.739818 9714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43881
I20260928 21:12:57.739850 9841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43881 every 8 connection(s)
I20260928 21:12:57.740715 9714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:57.744314 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9714
I20260928 21:12:57.744448 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:57.744797 392 external_mini_cluster.cc:1428] Setting key 4dc50b022dd4dc0ac7649b796645307f
I20260928 21:12:57.747006 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--builtin_ntp_servers=127.0.98.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:12:57.751327 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:57.742498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33081 (local address 127.0.98.62:44901)
0928 21:12:57.742638 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:57.742642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:57.743715 (+ 1073us) server_negotiation.cc:407] Connection header received
0928 21:12:57.745556 (+ 1841us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:57.745560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:57.745617 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:57.745700 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:57.746844 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.747479 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.749022 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.749412 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.749534 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.750697 (+ 1163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:57.750704 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:57.750711 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:57.750729 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:57.750767 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:57.750781 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:57.750872 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:57.751031 (+ 159us) server_negotiation.cc:299] Negotiation successful
0928 21:12:57.751113 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:12:57.752094 9844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:57.742912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44901 (local address 127.0.98.1:33081)
0928 21:12:57.743562 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:57.743606 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:57.745355 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:57.745916 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:57.745925 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:57.746033 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:57.746686 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:57.746705 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.747605 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.747612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:57.748891 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:57.748906 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.749571 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.749574 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:57.749718 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.750364 (+ 646us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:57.750366 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:57.750412 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:57.750502 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:57.750871 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:57.750878 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:57.750882 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:12:57.750990 (+ 108us) client_negotiation.cc:240] Negotiation successful
0928 21:12:57.751226 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":114,"threads_started":1}
I20260928 21:12:57.753572 9842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44901
I20260928 21:12:57.753875 9842 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:57.754581 9842 heartbeater.cc:511] Master 127.0.98.62:44901 requested a full tablet report, sending...
I20260928 21:12:57.756304 9654 ts_manager.cc:194] Registered new tserver with Master: 7fbc03238343471584abefc62927b1c8 (127.0.98.1:43881)
I20260928 21:12:57.757652 9654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:33081
W20260928 21:12:57.858399 9845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:57.858637 9845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:57.858672 9845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:57.862663 9845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:57.862718 9845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:57.862769 9845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:57.867540 9845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:57.868584 9845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:57.869928 9845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:57.879169 9850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:57.879343 9845 server_base.cc:1034] running on GCE node
W20260928 21:12:57.879208 9853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:57.879176 9851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:57.879990 9845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:57.880656 9845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:57.881809 9845 hybrid_clock.cc:648] HybridClock initialized: now 1790629977881778 us; error 54 us; skew 500 ppm
I20260928 21:12:57.883745 9845 webserver.cc:460] Webserver started at http://127.0.98.2:39111/ using document root <none> and password file <none>
I20260928 21:12:57.884311 9845 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:57.884372 9845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:57.884621 9845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:57.886675 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "9cc6742d2d1b4dbd8028bab6516a8262"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "994873361536f87ba55d59d79cb5ac26"
server_key_iv: "587ef42d22bdef00402f89c83763d0db"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.887212 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9cc6742d2d1b4dbd8028bab6516a8262"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "994873361536f87ba55d59d79cb5ac26"
server_key_iv: "587ef42d22bdef00402f89c83763d0db"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.890748 9845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:12:57.893321 9859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.894601 9845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:57.894748 9845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "9cc6742d2d1b4dbd8028bab6516a8262"
format_stamp: "Formatted at 2026-09-28 21:12:57 on dist-test-slave-v7xz"
server_key: "994873361536f87ba55d59d79cb5ac26"
server_key_iv: "587ef42d22bdef00402f89c83763d0db"
server_key_version: "encryptionkey@0"
I20260928 21:12:57.894869 9845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:57.910545 9845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:57.911204 9845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:57.911414 9845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:57.911978 9845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:57.912949 9845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:57.913015 9845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.913087 9845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:57.913120 9845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:57.921931 9845 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44481
I20260928 21:12:57.921955 9972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44481 every 8 connection(s)
I20260928 21:12:57.922833 9845 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:57.923347 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9845
I20260928 21:12:57.923468 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:57.923799 392 external_mini_cluster.cc:1428] Setting key b362591c3f1cd2518f7773fdb69f860c
I20260928 21:12:57.925976 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--builtin_ntp_servers=127.0.98.20:43861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260928 21:12:57.933208 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:57.924918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35089 (local address 127.0.98.62:44901)
0928 21:12:57.925056 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:57.925060 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:57.926439 (+ 1379us) server_negotiation.cc:407] Connection header received
0928 21:12:57.928239 (+ 1800us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:57.928244 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:57.928309 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:57.928435 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:57.929449 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.929968 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.931142 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.931365 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.931484 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.932659 (+ 1175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:57.932662 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:57.932665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:57.932685 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:57.932704 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:57.932712 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:57.932791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:57.932997 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:12:57.933061 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:12:57.934118 9975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:57.925471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44901 (local address 127.0.98.2:35089)
0928 21:12:57.926269 (+ 798us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:57.926328 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:57.928010 (+ 1682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:57.928612 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:57.928619 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:57.928707 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:57.929295 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:57.929310 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.930106 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.930109 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:57.931014 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:57.931023 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:57.931538 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:57.931542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:57.931719 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:57.932375 (+ 656us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:57.932376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:57.932421 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:57.932487 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:57.932807 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:57.932815 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:57.932821 (+ 6us) client_negotiation.cc:769] Sending connection context
0928 21:12:57.932997 (+ 176us) client_negotiation.cc:240] Negotiation successful
0928 21:12:57.933287 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":116,"threads_started":1}
I20260928 21:12:57.935432 9973 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44901
I20260928 21:12:57.935711 9973 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:57.936419 9973 heartbeater.cc:511] Master 127.0.98.62:44901 requested a full tablet report, sending...
I20260928 21:12:57.937801 9654 ts_manager.cc:194] Registered new tserver with Master: 9cc6742d2d1b4dbd8028bab6516a8262 (127.0.98.2:44481)
I20260928 21:12:57.938540 9654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:35089
W20260928 21:12:58.042237 9976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:58.042529 9976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:58.042567 9976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:58.046912 9976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:58.046990 9976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:58.047048 9976 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:58.052029 9976 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9976
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:58.053197 9976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:58.054544 9976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:58.064421 9984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.064417 9982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.064391 9981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:58.064728 9976 server_base.cc:1034] running on GCE node
I20260928 21:12:58.065326 9976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:58.066121 9976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:58.067317 9976 hybrid_clock.cc:648] HybridClock initialized: now 1790629978067280 us; error 49 us; skew 500 ppm
I20260928 21:12:58.069368 9976 webserver.cc:460] Webserver started at http://127.0.98.3:33531/ using document root <none> and password file <none>
I20260928 21:12:58.069978 9976 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:58.070043 9976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:58.070258 9976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:58.072395 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "6d7b34326fb64e58a9845f86849932dc"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "2759a0ace8bb76579f9a90c993388f6d"
server_key_iv: "1ac3d5922c891c00b1cdd60f68683ecf"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.073035 9976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "6d7b34326fb64e58a9845f86849932dc"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "2759a0ace8bb76579f9a90c993388f6d"
server_key_iv: "1ac3d5922c891c00b1cdd60f68683ecf"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.076813 9976 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:12:58.079494 9990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.080785 9976 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:12:58.080960 9976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "6d7b34326fb64e58a9845f86849932dc"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "2759a0ace8bb76579f9a90c993388f6d"
server_key_iv: "1ac3d5922c891c00b1cdd60f68683ecf"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.081092 9976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:58.100534 9976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:58.101361 9976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:58.101570 9976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:58.102192 9976 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:58.103185 9976 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:58.103262 9976 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.103329 9976 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:58.103382 9976 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.112485 9976 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45501
I20260928 21:12:58.112541 10103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45501 every 8 connection(s)
I20260928 21:12:58.113523 9976 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:58.123065 9693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.115474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59103 (local address 127.0.98.62:44901)
0928 21:12:58.115623 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.115628 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.116796 (+ 1168us) server_negotiation.cc:407] Connection header received
0928 21:12:58.118198 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.118201 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.118249 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.118329 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.119270 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:58.119971 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:58.121255 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:12:58.121492 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:12:58.121577 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:58.122636 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:58.122639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:58.122641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:58.122654 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:58.122673 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:58.122681 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:58.122741 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:58.122884 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:12:58.122942 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:12:58.123246 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 9976
I20260928 21:12:58.123351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:58.123646 392 external_mini_cluster.cc:1428] Setting key 0d738a86c2915c7db5b0bae3b912a547
I20260928 21:12:58.124220 10106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.115955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44901 (local address 127.0.98.3:59103)
0928 21:12:58.116639 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:58.116679 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:58.118003 (+ 1324us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:58.118502 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:58.118508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:58.118596 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:58.119138 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:58.119155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:58.120084 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:58.120087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:58.121143 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:12:58.121153 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:12:58.121634 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:12:58.121637 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:12:58.121787 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:12:58.122375 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:58.122376 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:58.122418 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:58.122488 (+ 70us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:58.122755 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:58.122761 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:58.122765 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:12:58.122864 (+ 99us) client_negotiation.cc:240] Negotiation successful
0928 21:12:58.123174 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":119,"threads_started":1}
I20260928 21:12:58.125649 10104 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44901
I20260928 21:12:58.125983 10104 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:58.126549 10104 heartbeater.cc:511] Master 127.0.98.62:44901 requested a full tablet report, sending...
I20260928 21:12:58.127558 9654 ts_manager.cc:194] Registered new tserver with Master: 6d7b34326fb64e58a9845f86849932dc (127.0.98.3:45501)
I20260928 21:12:58.128132 9654 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59103
I20260928 21:12:58.138283 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:12:58.141381 9693 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:58.140301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46370 (local address 127.0.98.62:44901)
0928 21:12:58.140453 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.140457 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.140641 (+ 184us) server_negotiation.cc:407] Connection header received
0928 21:12:58.140736 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.140738 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.140779 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.140917 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.141216 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46370: BlockingRecv error: recv got EOF from 127.0.0.1:46370 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:12:58.141633 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9714
I20260928 21:12:58.147547 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9845
I20260928 21:12:58.153012 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9976
I20260928 21:12:58.158305 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 9623
2026-09-28T21:12:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (873 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-28T21:12:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:12:58Z Disabled control of system clock
I20260928 21:12:58.187537 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35911
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35911
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:12:58.295773 10115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:58.296026 10115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:58.296062 10115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:58.299889 10115 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:12:58.299937 10115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:58.299952 10115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:58.299968 10115 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:12:58.299983 10115 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:12:58.304557 10115 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35911
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35911
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:58.305565 10115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:58.306834 10115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:12:58.315526 10115 server_base.cc:1034] running on GCE node
W20260928 21:12:58.315450 10123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.315438 10121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.315433 10120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:58.316475 10115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:58.317818 10115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:58.319031 10115 hybrid_clock.cc:648] HybridClock initialized: now 1790629978319017 us; error 31 us; skew 500 ppm
I20260928 21:12:58.321200 10115 webserver.cc:460] Webserver started at http://127.0.98.62:43745/ using document root <none> and password file <none>
I20260928 21:12:58.321856 10115 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:58.321962 10115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:58.322222 10115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:58.324465 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "caada4ce8d4d45b0824af678aac839f6"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "9ceb68511a6306ba9dda58762676a3af"
server_key_iv: "52b151d53d9acc07ca6fb80790eab46a"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.325112 10115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "caada4ce8d4d45b0824af678aac839f6"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "9ceb68511a6306ba9dda58762676a3af"
server_key_iv: "52b151d53d9acc07ca6fb80790eab46a"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.329190 10115 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:12:58.331845 10129 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.333297 10115 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:58.333442 10115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "caada4ce8d4d45b0824af678aac839f6"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "9ceb68511a6306ba9dda58762676a3af"
server_key_iv: "52b151d53d9acc07ca6fb80790eab46a"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.333623 10115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:58.338836 10115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:58.339468 10115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:58.339660 10115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:58.347043 10115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35911
I20260928 21:12:58.347059 10181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35911 every 8 connection(s)
I20260928 21:12:58.348061 10115 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:12:58.351264 10182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:12:58.353441 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10115
I20260928 21:12:58.353562 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:12:58.353860 392 external_mini_cluster.cc:1428] Setting key b6c1427b30492c90b7f0725c0c5c8985
I20260928 21:12:58.357822 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6: Bootstrap starting.
I20260928 21:12:58.359881 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.355470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38266 (local address 127.0.98.62:35911)
0928 21:12:58.356158 (+ 688us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.356173 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.356212 (+ 39us) server_negotiation.cc:407] Connection header received
0928 21:12:58.357063 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.357083 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.357454 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.357846 (+ 392us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.358584 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:58.358589 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:58.358601 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:58.358610 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:58.358660 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:58.358676 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:58.358848 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:58.358924 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:12:58.359164 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":142,"threads_started":1}
I20260928 21:12:58.360436 10182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6: Neither blocks nor log segments found. Creating new log.
I20260928 21:12:58.361346 10182 log.cc:824] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:12:58.363307 10182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6: No bootstrap required, opened a new log
I20260928 21:12:58.366832 10182 raft_consensus.cc:348] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caada4ce8d4d45b0824af678aac839f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35911 } }
I20260928 21:12:58.367115 10182 raft_consensus.cc:374] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:12:58.367172 10182 raft_consensus.cc:729] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: caada4ce8d4d45b0824af678aac839f6, State: Initialized, Role: FOLLOWER
I20260928 21:12:58.367651 10182 consensus_queue.cc:260] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caada4ce8d4d45b0824af678aac839f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35911 } }
I20260928 21:12:58.367771 10182 raft_consensus.cc:388] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:12:58.367852 10182 raft_consensus.cc:482] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:12:58.367936 10182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:12:58.368810 10182 raft_consensus.cc:504] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caada4ce8d4d45b0824af678aac839f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35911 } }
I20260928 21:12:58.369194 10182 leader_election.cc:302] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: caada4ce8d4d45b0824af678aac839f6; no voters:
I20260928 21:12:58.369486 10182 leader_election.cc:288] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:12:58.369635 10187 raft_consensus.cc:2781] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:12:58.369879 10187 raft_consensus.cc:686] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [term 1 LEADER]: Becoming Leader. State: Replica: caada4ce8d4d45b0824af678aac839f6, State: Running, Role: LEADER
I20260928 21:12:58.370198 10187 consensus_queue.cc:237] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caada4ce8d4d45b0824af678aac839f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35911 } }
I20260928 21:12:58.370674 10182 sys_catalog.cc:564] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:12:58.371793 10188 sys_catalog.cc:455] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "caada4ce8d4d45b0824af678aac839f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caada4ce8d4d45b0824af678aac839f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35911 } } }
I20260928 21:12:58.371912 10188 sys_catalog.cc:458] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:58.372186 10189 sys_catalog.cc:455] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader caada4ce8d4d45b0824af678aac839f6. Latest consensus state: current_term: 1 leader_uuid: "caada4ce8d4d45b0824af678aac839f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "caada4ce8d4d45b0824af678aac839f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35911 } } }
I20260928 21:12:58.372284 10189 sys_catalog.cc:458] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6 [sys.catalog]: This master's current role is: LEADER
I20260928 21:12:58.372397 10196 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:12:58.375340 10196 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:12:58.381273 10196 catalog_manager.cc:1347] Generated new cluster ID: 8311d911b7094e8f8d685f55e317c113
I20260928 21:12:58.381354 10196 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:12:58.396363 10196 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:12:58.396737 10196 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:12:58.413698 10196 catalog_manager.cc:5868] T 00000000000000000000000000000000 P caada4ce8d4d45b0824af678aac839f6: Generated new TSK 0
I20260928 21:12:58.414495 10196 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:12:58.419418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35911
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260928 21:12:58.534639 10206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:58.534969 10206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:58.535065 10206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:58.539659 10206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:58.539736 10206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:58.539798 10206 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:12:58.544755 10206 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:58.545989 10206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:58.547359 10206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:58.556955 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:58.557125 10206 server_base.cc:1034] running on GCE node
W20260928 21:12:58.557006 10211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.556955 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:58.557672 10206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:58.558346 10206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:58.559543 10206 hybrid_clock.cc:648] HybridClock initialized: now 1790629978559496 us; error 64 us; skew 500 ppm
I20260928 21:12:58.561626 10206 webserver.cc:460] Webserver started at http://127.0.98.1:43101/ using document root <none> and password file <none>
I20260928 21:12:58.562259 10206 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:58.562361 10206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:58.562608 10206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:58.564759 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "1271ceacc2764a098ed58fa0015cbc39"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "c24816ad9fef633a8f4b131699531a96"
server_key_iv: "09748d6187c4d862e0a5dcefdbeb8dff"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.565416 10206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "1271ceacc2764a098ed58fa0015cbc39"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "c24816ad9fef633a8f4b131699531a96"
server_key_iv: "09748d6187c4d862e0a5dcefdbeb8dff"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.569265 10206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:58.571995 10220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.573812 10206 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:12:58.573956 10206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "1271ceacc2764a098ed58fa0015cbc39"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "c24816ad9fef633a8f4b131699531a96"
server_key_iv: "09748d6187c4d862e0a5dcefdbeb8dff"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.574097 10206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:58.580781 10206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:58.581593 10206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:58.581826 10206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:58.582542 10206 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:58.583484 10206 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:58.583531 10206 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.583600 10206 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:58.583630 10206 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.592505 10206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44923
I20260928 21:12:58.592530 10333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44923 every 8 connection(s)
I20260928 21:12:58.593483 10206 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:12:58.596308 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10206
I20260928 21:12:58.596429 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:12:58.596737 392 external_mini_cluster.cc:1428] Setting key e8623c87b5c54910a561393cb37930bc
I20260928 21:12:58.598984 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35911
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:12:58.600611 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.595373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50263 (local address 127.0.98.62:35911)
0928 21:12:58.595549 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.595554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.596886 (+ 1332us) server_negotiation.cc:407] Connection header received
0928 21:12:58.598500 (+ 1614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.598503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.598537 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.598610 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.600055 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:58.600058 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:58.600061 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:58.600073 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:58.600092 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:58.600100 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:58.600181 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:58.600383 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:12:58.600436 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260928 21:12:58.601531 10336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.595845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35911 (local address 127.0.98.1:50263)
0928 21:12:58.596659 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:58.596718 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:58.598229 (+ 1511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:58.598804 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:58.598816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:58.598954 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:58.599720 (+ 766us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:58.599724 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:58.599785 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:58.599884 (+ 99us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:58.600193 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:58.600204 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:58.600214 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:12:58.600404 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 21:12:58.600691 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":111,"threads_started":1}
I20260928 21:12:58.602774 10334 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35911
I20260928 21:12:58.603026 10334 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:58.603439 10334 heartbeater.cc:511] Master 127.0.98.62:35911 requested a full tablet report, sending...
I20260928 21:12:58.604966 10146 ts_manager.cc:194] Registered new tserver with Master: 1271ceacc2764a098ed58fa0015cbc39 (127.0.98.1:44923)
W20260928 21:12:58.715916 10337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:58.716223 10337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:58.716270 10337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:58.720697 10337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:58.720767 10337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:58.720889 10337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:12:58.725972 10337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:58.727169 10337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:58.728628 10337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:58.738552 10345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.738575 10342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.738575 10343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:58.739166 10337 server_base.cc:1034] running on GCE node
I20260928 21:12:58.739580 10337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:58.740221 10337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:58.741374 10337 hybrid_clock.cc:648] HybridClock initialized: now 1790629978741340 us; error 48 us; skew 500 ppm
I20260928 21:12:58.743216 10337 webserver.cc:460] Webserver started at http://127.0.98.2:37535/ using document root <none> and password file <none>
I20260928 21:12:58.743764 10337 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:58.743825 10337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:58.743991 10337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:58.746145 10337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "c17966581df04aa495279dc096697d45"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "e40539a7f91c16ae487edea5a06325e7"
server_key_iv: "59bac641a0d268f4682304eea8c59017"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.746703 10337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c17966581df04aa495279dc096697d45"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "e40539a7f91c16ae487edea5a06325e7"
server_key_iv: "59bac641a0d268f4682304eea8c59017"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.750542 10337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:12:58.753006 10351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.754264 10337 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:12:58.754361 10337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "c17966581df04aa495279dc096697d45"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "e40539a7f91c16ae487edea5a06325e7"
server_key_iv: "59bac641a0d268f4682304eea8c59017"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.754463 10337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:58.758987 10337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:58.759577 10337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:58.759714 10337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:58.760202 10337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:58.761152 10337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:58.761201 10337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.761240 10337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:58.761256 10337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.770478 10337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33855
I20260928 21:12:58.770495 10464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33855 every 8 connection(s)
I20260928 21:12:58.771543 10337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:12:58.775876 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10337
I20260928 21:12:58.776010 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:12:58.776343 392 external_mini_cluster.cc:1428] Setting key ce2f138dd3363c846254f48f8a490fcd
I20260928 21:12:58.778092 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.773523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36861 (local address 127.0.98.62:35911)
0928 21:12:58.773658 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.773662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.774756 (+ 1094us) server_negotiation.cc:407] Connection header received
0928 21:12:58.776063 (+ 1307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.776066 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.776111 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.776209 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.777555 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:58.777558 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:58.777560 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:58.777572 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:58.777590 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:58.777598 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:58.777710 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:58.777886 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 21:12:58.777929 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:12:58.778509 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35911
--builtin_ntp_servers=127.0.98.20:35557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260928 21:12:58.779176 10467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.773942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35911 (local address 127.0.98.2:36861)
0928 21:12:58.774601 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:58.774642 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:58.775809 (+ 1167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:58.776377 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:58.776434 (+ 57us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:58.776543 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:58.777253 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:58.777255 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:58.777309 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:58.777395 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:58.777712 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:58.777721 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:58.777730 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:12:58.777927 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:12:58.778163 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":109,"threads_started":1}
I20260928 21:12:58.780433 10465 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35911
I20260928 21:12:58.780685 10465 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:58.781200 10465 heartbeater.cc:511] Master 127.0.98.62:35911 requested a full tablet report, sending...
I20260928 21:12:58.782411 10146 ts_manager.cc:194] Registered new tserver with Master: c17966581df04aa495279dc096697d45 (127.0.98.2:33855)
W20260928 21:12:58.891609 10468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:12:58.891880 10468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:12:58.891921 10468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:12:58.896076 10468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:12:58.896131 10468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:12:58.896236 10468 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:12:58.901039 10468 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35911
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:12:58.902180 10468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:12:58.903551 10468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:12:58.913607 10476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.913651 10473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:12:58.913641 10474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:12:58.913880 10468 server_base.cc:1034] running on GCE node
I20260928 21:12:58.914314 10468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:12:58.915000 10468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:12:58.916194 10468 hybrid_clock.cc:648] HybridClock initialized: now 1790629978916148 us; error 65 us; skew 500 ppm
I20260928 21:12:58.918358 10468 webserver.cc:460] Webserver started at http://127.0.98.3:45245/ using document root <none> and password file <none>
I20260928 21:12:58.918987 10468 fs_manager.cc:359] Metadata directory not provided
I20260928 21:12:58.919081 10468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:12:58.919324 10468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:12:58.921716 10468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "2ff8804d0a7b4ae5aa6b1341e6aaf087"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "5c0f57d6f32f8d1a6b934e7a76970571"
server_key_iv: "c4066dc53949032fa8604a9ecd0c6756"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.922348 10468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2ff8804d0a7b4ae5aa6b1341e6aaf087"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "5c0f57d6f32f8d1a6b934e7a76970571"
server_key_iv: "c4066dc53949032fa8604a9ecd0c6756"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.926272 10468 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:12:58.928776 10482 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.930106 10468 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:12:58.930244 10468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "2ff8804d0a7b4ae5aa6b1341e6aaf087"
format_stamp: "Formatted at 2026-09-28 21:12:58 on dist-test-slave-v7xz"
server_key: "5c0f57d6f32f8d1a6b934e7a76970571"
server_key_iv: "c4066dc53949032fa8604a9ecd0c6756"
server_key_version: "encryptionkey@0"
I20260928 21:12:58.930368 10468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:12:58.935168 10468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:12:58.935781 10468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:12:58.935966 10468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:12:58.936609 10468 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:12:58.937659 10468 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:12:58.937728 10468 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.937795 10468 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:12:58.937844 10468 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:12:58.946753 10468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34365
I20260928 21:12:58.946786 10595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34365 every 8 connection(s)
I20260928 21:12:58.947701 10468 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:12:58.954129 10185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.949680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53791 (local address 127.0.98.62:35911)
0928 21:12:58.949835 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.949839 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.950939 (+ 1100us) server_negotiation.cc:407] Connection header received
0928 21:12:58.952254 (+ 1315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.952258 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.952306 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.952414 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.953691 (+ 1277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:12:58.953694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:12:58.953696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:12:58.953708 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:12:58.953748 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:12:58.953764 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:12:58.953854 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:12:58.953984 (+ 130us) server_negotiation.cc:299] Negotiation successful
0928 21:12:58.954023 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260928 21:12:58.955025 10598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:12:58.950107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35911 (local address 127.0.98.3:53791)
0928 21:12:58.950767 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0928 21:12:58.950810 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:12:58.952027 (+ 1217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:12:58.952594 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:12:58.952607 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:12:58.952701 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:12:58.953339 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:12:58.953341 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:12:58.953382 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:12:58.953507 (+ 125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:12:58.953850 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:12:58.953856 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:12:58.953861 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:12:58.954003 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 21:12:58.954236 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":112,"threads_started":1}
I20260928 21:12:58.955386 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10468
I20260928 21:12:58.955513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:12:58.955801 392 external_mini_cluster.cc:1428] Setting key 76257dfcd905a73041b964505cbd2f5b
I20260928 21:12:58.956292 10596 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35911
I20260928 21:12:58.956547 10596 heartbeater.cc:464] Registering TS with master...
I20260928 21:12:58.957070 10596 heartbeater.cc:511] Master 127.0.98.62:35911 requested a full tablet report, sending...
I20260928 21:12:58.958094 10146 ts_manager.cc:194] Registered new tserver with Master: 2ff8804d0a7b4ae5aa6b1341e6aaf087 (127.0.98.3:34365)
I20260928 21:12:58.969848 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260928 21:12:58.973023 10185 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:12:58.971908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.0.98.62:35911)
0928 21:12:58.972051 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:12:58.972056 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:12:58.972303 (+ 247us) server_negotiation.cc:407] Connection header received
0928 21:12:58.972462 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:12:58.972465 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:12:58.972509 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:12:58.972599 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:12:58.972857 (+ 258us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38282: BlockingRecv error: recv got EOF from 127.0.0.1:38282 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:12:58.973266 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10206
I20260928 21:12:58.978508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10337
I20260928 21:12:58.983942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10468
I20260928 21:12:58.988988 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10115
2026-09-28T21:12:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (830 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:12:59 dist-test-slave-v7xz krb5kdc[10606](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:12:59 dist-test-slave-v7xz krb5kdc[10606](info): set up 2 sockets
Sep 28 21:12:59 dist-test-slave-v7xz krb5kdc[10606](info): commencing operation
krb5kdc: starting...
W20260928 21:13:00.074369 392 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:13:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:13:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:13:00.195808 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44749
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44749
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:00.310103 10619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:00.310422 10619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:00.310510 10619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:00.314497 10619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:13:00.314569 10619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:00.314589 10619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:00.314606 10619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:13:00.314621 10619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:13:00.319902 10619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44749
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44749
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:00.320933 10619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:00.321894 10619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:13:00.330542 10619 server_base.cc:1034] running on GCE node
W20260928 21:13:00.330469 10624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:00.330469 10627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:00.330492 10625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:00.331425 10619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:00.332500 10619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:00.333673 10619 hybrid_clock.cc:648] HybridClock initialized: now 1790629980333628 us; error 57 us; skew 500 ppm
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:00.341753 10619 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:13:00.342686 10619 webserver.cc:460] Webserver started at http://127.0.98.62:45343/ using document root <none> and password file <none>
I20260928 21:13:00.343243 10619 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:00.343304 10619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:00.343472 10619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:13:00.345629 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "552d4b72c5d8431aa9a86f1c0d811e54"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "30f02cf164f16823b254cc50337a1985"
server_key_iv: "5c6959a7379afad71ab324e43d389570"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.346148 10619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "552d4b72c5d8431aa9a86f1c0d811e54"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "30f02cf164f16823b254cc50337a1985"
server_key_iv: "5c6959a7379afad71ab324e43d389570"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.349961 10619 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:13:00.352378 10634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.353758 10619 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:13:00.353892 10619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "552d4b72c5d8431aa9a86f1c0d811e54"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "30f02cf164f16823b254cc50337a1985"
server_key_iv: "5c6959a7379afad71ab324e43d389570"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.353998 10619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:00.375650 10619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:00.378973 10619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:00.379199 10619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:00.386479 10619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44749
I20260928 21:13:00.386773 10686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44749 every 8 connection(s)
I20260928 21:13:00.387485 10619 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:13:00.390233 10687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:00.392149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10619
I20260928 21:13:00.392274 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:13:00.392539 392 external_mini_cluster.cc:1428] Setting key 1ada06db4edb4209987ee67a195033af
I20260928 21:13:00.396564 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54: Bootstrap starting.
I20260928 21:13:00.398789 10687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:00.399495 10687 log.cc:824] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:13:00.401388 10687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54: No bootstrap required, opened a new log
I20260928 21:13:00.405019 10687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44749 } }
I20260928 21:13:00.405290 10687 raft_consensus.cc:374] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:00.405377 10687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 552d4b72c5d8431aa9a86f1c0d811e54, State: Initialized, Role: FOLLOWER
I20260928 21:13:00.405835 10687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44749 } }
I20260928 21:13:00.405992 10687 raft_consensus.cc:388] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:13:00.406078 10687 raft_consensus.cc:482] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:13:00.406177 10687 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:00.406205 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:00.393961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.0.98.62:44749)
0928 21:13:00.394435 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:00.394450 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:00.394486 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:13:00.395107 (+ 621us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:00.395125 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:00.395451 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:00.395783 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:00.396547 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.397610 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.398448 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.398947 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.399123 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:00.401532 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:00.401537 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:00.401549 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:00.401572 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:00.403322 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:00.403753 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:00.403759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:00.403765 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:00.403879 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:00.404155 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:00.404159 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:00.404161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:00.404531 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:00.404742 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:00.405043 (+ 301us) server_negotiation.cc:299] Negotiation successful
0928 21:13:00.405369 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":128,"threads_started":1}
I20260928 21:13:00.407270 10687 raft_consensus.cc:504] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44749 } }
I20260928 21:13:00.407615 10687 leader_election.cc:302] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 552d4b72c5d8431aa9a86f1c0d811e54; no voters:
I20260928 21:13:00.407945 10687 leader_election.cc:288] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:13:00.408008 10692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:13:00.408205 10692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [term 1 LEADER]: Becoming Leader. State: Replica: 552d4b72c5d8431aa9a86f1c0d811e54, State: Running, Role: LEADER
I20260928 21:13:00.408850 10692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44749 } }
I20260928 21:13:00.409204 10687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:13:00.410662 10694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44749 } } }
I20260928 21:13:00.410784 10694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [sys.catalog]: This master's current role is: LEADER
I20260928 21:13:00.411175 10704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:13:00.411680 10693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 552d4b72c5d8431aa9a86f1c0d811e54. Latest consensus state: current_term: 1 leader_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "552d4b72c5d8431aa9a86f1c0d811e54" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44749 } } }
I20260928 21:13:00.412010 10693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54 [sys.catalog]: This master's current role is: LEADER
I20260928 21:13:00.413874 10704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:13:00.419309 10704 catalog_manager.cc:1347] Generated new cluster ID: 03d3661bfd9e4bbb9a973ad97c001e1e
I20260928 21:13:00.419385 10704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:13:00.428005 10704 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:13:00.429329 10704 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:13:00.439615 10704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54: Generated new TSK 0
I20260928 21:13:00.440450 10704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:13:00.471956 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44749
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:00.588670 10713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:00.589010 10713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:00.589059 10713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:00.593050 10713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:00.593104 10713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:00.593163 10713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:13:00.598349 10713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:00.599335 10713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:00.600265 10713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:00.609465 10718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:00.609540 10721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:00.609485 10719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:00.609931 10713 server_base.cc:1034] running on GCE node
I20260928 21:13:00.610394 10713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:00.611028 10713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:00.612217 10713 hybrid_clock.cc:648] HybridClock initialized: now 1790629980612173 us; error 62 us; skew 500 ppm
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:00.615098 10713 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:13:00.615996 10713 webserver.cc:460] Webserver started at http://127.0.98.1:35765/ using document root <none> and password file <none>
I20260928 21:13:00.616554 10713 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:00.616642 10713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:00.616940 10713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:13:00.619000 10713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "cf9ac4436e524db7a5868d9a056b3093"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "89393dc3c51ec7caf7fbaeb7a2a5c45f"
server_key_iv: "bd25ca12bde671a3d0e918fdf67a740e"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.619562 10713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cf9ac4436e524db7a5868d9a056b3093"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "89393dc3c51ec7caf7fbaeb7a2a5c45f"
server_key_iv: "bd25ca12bde671a3d0e918fdf67a740e"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.623189 10713 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 21:13:00.625525 10728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.626777 10713 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:13:00.626924 10713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "cf9ac4436e524db7a5868d9a056b3093"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "89393dc3c51ec7caf7fbaeb7a2a5c45f"
server_key_iv: "bd25ca12bde671a3d0e918fdf67a740e"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.627034 10713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:00.658144 10713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:00.661664 10713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:00.661886 10713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:00.662514 10713 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:13:00.663371 10713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:13:00.663450 10713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.663524 10713 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:13:00.663578 10713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.672504 10713 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45911
I20260928 21:13:00.672528 10841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45911 every 8 connection(s)
I20260928 21:13:00.673470 10713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:13:00.678577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10713
I20260928 21:13:00.678701 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:13:00.679023 392 external_mini_cluster.cc:1428] Setting key a31317e9ef34ede0ddd1849d888fee75
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:13:00.687968 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:00.675343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54751 (local address 127.0.98.62:44749)
0928 21:13:00.675487 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:00.675491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:00.676530 (+ 1039us) server_negotiation.cc:407] Connection header received
0928 21:13:00.677489 (+ 959us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:00.677492 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:00.677537 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:00.677615 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:00.679036 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.679787 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.681398 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.681620 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.681704 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:00.684365 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:00.684368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:00.684371 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:00.684397 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:00.685928 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:00.686460 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:00.686465 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:00.686467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:00.686549 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:00.686898 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:00.686903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:00.686905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:00.687088 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:00.687198 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:00.687685 (+ 487us) server_negotiation.cc:299] Negotiation successful
0928 21:13:00.687804 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:13:00.688859 10844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:00.675759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44749 (local address 127.0.98.1:54751)
0928 21:13:00.676401 (+ 642us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:00.676443 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:00.677276 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:00.677794 (+ 518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:00.677801 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:00.678163 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:13:00.678858 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:00.678879 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.679910 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.679913 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:00.681261 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:00.681275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.681771 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.681778 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:00.682016 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:00.682713 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:13:00.682729 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:13:00.684188 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:13:00.686058 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:00.686065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:00.686081 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:00.686353 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:00.686650 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:00.686653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:00.686655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:00.686796 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:00.687195 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:13:00.687202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:13:00.687457 (+ 255us) client_negotiation.cc:769] Sending connection context
0928 21:13:00.687647 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 21:13:00.687989 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":117,"threads_started":1}
I20260928 21:13:00.690163 10842 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44749
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:13:00.690435 10842 heartbeater.cc:464] Registering TS with master...
I20260928 21:13:00.690938 10842 heartbeater.cc:511] Master 127.0.98.62:44749 requested a full tablet report, sending...
I20260928 21:13:00.692641 10651 ts_manager.cc:194] Registered new tserver with Master: cf9ac4436e524db7a5868d9a056b3093 (127.0.98.1:45911)
I20260928 21:13:00.694370 10651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54751
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:13:00.707415 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44749
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:00.819269 10847 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:00.819541 10847 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:00.819585 10847 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:00.823676 10847 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:00.823740 10847 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:00.823832 10847 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:13:00.829166 10847 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10847
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:00.830302 10847 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:00.831430 10847 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:00.841225 10853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:00.841183 10852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:00.841192 10855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:00.841475 10847 server_base.cc:1034] running on GCE node
I20260928 21:13:00.841938 10847 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:00.842562 10847 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:00.843720 10847 hybrid_clock.cc:648] HybridClock initialized: now 1790629980843704 us; error 34 us; skew 500 ppm
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:00.846565 10847 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:13:00.847461 10847 webserver.cc:460] Webserver started at http://127.0.98.2:43443/ using document root <none> and password file <none>
I20260928 21:13:00.848008 10847 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:00.848066 10847 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:00.848280 10847 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:13:00.850338 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "b61713c5a7f347b3a5a7ccf3c116261e"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "9050d5e7d0025dabc6bc391945b41341"
server_key_iv: "75c1710293668899d26e0c358f90e7a6"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.850883 10847 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b61713c5a7f347b3a5a7ccf3c116261e"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "9050d5e7d0025dabc6bc391945b41341"
server_key_iv: "75c1710293668899d26e0c358f90e7a6"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.854619 10847 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:13:00.856958 10862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.858273 10847 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:13:00.858402 10847 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "b61713c5a7f347b3a5a7ccf3c116261e"
format_stamp: "Formatted at 2026-09-28 21:13:00 on dist-test-slave-v7xz"
server_key: "9050d5e7d0025dabc6bc391945b41341"
server_key_iv: "75c1710293668899d26e0c358f90e7a6"
server_key_version: "encryptionkey@0"
I20260928 21:13:00.858512 10847 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:00.877254 10847 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:00.880112 10847 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:00.880330 10847 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:00.880975 10847 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:13:00.881826 10847 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:13:00.881875 10847 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.881943 10847 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:13:00.881978 10847 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:00.890836 10847 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:38305
I20260928 21:13:00.890861 10975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:38305 every 8 connection(s)
I20260928 21:13:00.891744 10847 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:13:00.894326 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10847
I20260928 21:13:00.894479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:13:00.894783 392 external_mini_cluster.cc:1428] Setting key ba7affcdfa287781ec9613336f9e396b
Sep 28 21:13:00 dist-test-slave-v7xz krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:13:00.906400 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:00.893654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45877 (local address 127.0.98.62:44749)
0928 21:13:00.893783 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:00.893788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:00.894896 (+ 1108us) server_negotiation.cc:407] Connection header received
0928 21:13:00.895895 (+ 999us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:00.895898 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:00.895940 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:00.896050 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:00.897367 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.897883 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.899008 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.899235 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.899348 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:00.902185 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:00.902188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:00.902190 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:00.902212 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:00.903741 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:00.904474 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:00.904478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:00.904480 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:00.904569 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:00.904974 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:00.904977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:00.904978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:00.905265 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:00.905379 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:00.906121 (+ 742us) server_negotiation.cc:299] Negotiation successful
0928 21:13:00.906245 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:13:00.907181 10978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:00.894024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44749 (local address 127.0.98.2:45877)
0928 21:13:00.894746 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:00.894799 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:00.895675 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:00.896269 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:00.896279 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:00.896639 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:13:00.897188 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:00.897202 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.898007 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.898011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:00.898881 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:00.898891 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:00.899382 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:00.899387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:00.899551 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:00.900159 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:13:00.900174 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:13:00.901994 (+ 1820us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:13:00.903872 (+ 1878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:00.903881 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:00.903901 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:00.904256 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:00.904665 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:00.904668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:00.904669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:00.904805 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:00.905398 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:13:00.905406 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:13:00.905769 (+ 363us) client_negotiation.cc:769] Sending connection context
0928 21:13:00.905928 (+ 159us) client_negotiation.cc:240] Negotiation successful
0928 21:13:00.906206 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":146,"threads_started":1}
I20260928 21:13:00.908529 10976 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44749
I20260928 21:13:00.908881 10976 heartbeater.cc:464] Registering TS with master...
I20260928 21:13:00.909523 10976 heartbeater.cc:511] Master 127.0.98.62:44749 requested a full tablet report, sending...
I20260928 21:13:00.910770 10651 ts_manager.cc:194] Registered new tserver with Master: b61713c5a7f347b3a5a7ccf3c116261e (127.0.98.2:38305)
I20260928 21:13:00.911370 10651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:13:00.924180 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44749
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:01.036115 10981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:01.036397 10981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:01.036448 10981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:01.040560 10981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:01.040621 10981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:01.040683 10981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:13:01.046007 10981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44749
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:01.047067 10981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:01.048094 10981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:01.057740 10986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:01.057767 10989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:01.057757 10987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:01.058003 10981 server_base.cc:1034] running on GCE node
I20260928 21:13:01.058476 10981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:01.059119 10981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:01.060278 10981 hybrid_clock.cc:648] HybridClock initialized: now 1790629981060242 us; error 51 us; skew 500 ppm
Sep 28 21:13:01 dist-test-slave-v7xz krb5kdc[10606](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:01.063148 10981 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:13:01.064043 10981 webserver.cc:460] Webserver started at http://127.0.98.3:35689/ using document root <none> and password file <none>
I20260928 21:13:01.064563 10981 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:01.064620 10981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:01.064786 10981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:13:01.066824 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "359617309f0b407fa109909806fddf3a"
format_stamp: "Formatted at 2026-09-28 21:13:01 on dist-test-slave-v7xz"
server_key: "ffcf90dbf5534becc53d6a2d128f84ca"
server_key_iv: "c545a31ebac40cdf894597a1a3f708d4"
server_key_version: "encryptionkey@0"
I20260928 21:13:01.067374 10981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "359617309f0b407fa109909806fddf3a"
format_stamp: "Formatted at 2026-09-28 21:13:01 on dist-test-slave-v7xz"
server_key: "ffcf90dbf5534becc53d6a2d128f84ca"
server_key_iv: "c545a31ebac40cdf894597a1a3f708d4"
server_key_version: "encryptionkey@0"
I20260928 21:13:01.071156 10981 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:13:01.073860 10996 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:01.075062 10981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:13:01.075192 10981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "359617309f0b407fa109909806fddf3a"
format_stamp: "Formatted at 2026-09-28 21:13:01 on dist-test-slave-v7xz"
server_key: "ffcf90dbf5534becc53d6a2d128f84ca"
server_key_iv: "c545a31ebac40cdf894597a1a3f708d4"
server_key_version: "encryptionkey@0"
I20260928 21:13:01.075299 10981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:01.089028 10981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:01.092355 10981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:01.092571 10981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:01.093251 10981 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:13:01.094079 10981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:13:01.094151 10981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:01.094223 10981 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:13:01.094265 10981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:01.103229 10981 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46741
I20260928 21:13:01.103264 11109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46741 every 8 connection(s)
I20260928 21:13:01.104144 10981 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:13:01.110873 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 10981
I20260928 21:13:01.110984 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:13:01.111302 392 external_mini_cluster.cc:1428] Setting key d5e5baf1df7961c6ef17400738a5aee0
Sep 28 21:13:01 dist-test-slave-v7xz krb5kdc[10606](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:13:01.118712 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.106041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55943 (local address 127.0.98.62:44749)
0928 21:13:01.106211 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.106216 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.107288 (+ 1072us) server_negotiation.cc:407] Connection header received
0928 21:13:01.108157 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.108159 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.108204 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.108281 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:01.109687 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.110375 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.111873 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.112194 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.112292 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.115009 (+ 2717us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:01.115012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:01.115014 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:01.115038 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:01.116687 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:01.117299 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:01.117302 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:01.117303 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:01.117376 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:01.117726 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:01.117729 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:01.117730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:01.117901 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:01.117978 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.118494 (+ 516us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.118595 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:13:01.119609 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.106394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44749 (local address 127.0.98.3:55943)
0928 21:13:01.107132 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.107176 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:01.107977 (+ 801us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:01.108457 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:01.108463 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:01.109005 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:13:01.109553 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.109568 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.110485 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.110487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.111736 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.111761 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.112358 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.112362 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.112532 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.113197 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:13:01.113219 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:13:01.114854 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:13:01.116793 (+ 1939us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:01.116798 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:01.116809 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:01.117210 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:01.117487 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:01.117490 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:01.117491 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:01.117631 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:01.118014 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:13:01.118020 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:13:01.118263 (+ 243us) client_negotiation.cc:769] Sending connection context
0928 21:13:01.118431 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 21:13:01.118692 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":115,"threads_started":1}
I20260928 21:13:01.120931 11110 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44749
I20260928 21:13:01.121186 11110 heartbeater.cc:464] Registering TS with master...
I20260928 21:13:01.121694 11110 heartbeater.cc:511] Master 127.0.98.62:44749 requested a full tablet report, sending...
I20260928 21:13:01.122771 10651 ts_manager.cc:194] Registered new tserver with Master: 359617309f0b407fa109909806fddf3a (127.0.98.3:46741)
I20260928 21:13:01.123365 10651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55943
I20260928 21:13:01.125449 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:13:01.134895 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.127423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.0.98.62:44749)
0928 21:13:01.127577 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.127585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.127736 (+ 151us) server_negotiation.cc:407] Connection header received
0928 21:13:01.127825 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.127827 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.127864 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.127936 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:01.128797 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.129440 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.130391 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.130628 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.130721 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.132065 (+ 1344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:01.132068 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:01.132070 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:01.132090 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:01.133547 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:01.133928 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:01.133931 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:01.133932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:01.134004 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:01.134228 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:01.134230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:01.134231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:01.134434 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:01.134536 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.134688 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.134773 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:13:01.137801 10651 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54326:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:13:01.140735 10651 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:13:01.155308 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.149721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55784 (local address 127.0.98.3:46741)
0928 21:13:01.150929 (+ 1208us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.150938 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.150983 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:13:01.151051 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.151056 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.151254 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.151401 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:01.151669 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.152442 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.153933 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.154600 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.154718 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.154859 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.155048 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.155144 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1115,"thread_start_us":167,"threads_started":1}
I20260928 21:13:01.155294 11122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.149547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46741 (local address 127.0.0.1:55784)
0928 21:13:01.150606 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.150622 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:01.150722 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:01.151440 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:01.151442 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:01.151457 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:01.151558 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.151564 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.152587 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.152593 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.153813 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.153822 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.154728 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.154731 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.154846 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.154886 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:13:01.155005 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:13:01.155119 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":991,"thread_start_us":86,"threads_started":1}
I20260928 21:13:01.155952 11121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.149450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38305 (local address 127.0.0.1:53714)
0928 21:13:01.150273 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.150340 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:01.150473 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:01.151256 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:01.151262 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:01.151282 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:01.151434 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.151441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.152955 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.152958 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.154266 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.154275 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.155559 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.155563 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.155671 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.155694 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:13:01.155761 (+ 67us) client_negotiation.cc:240] Negotiation successful
0928 21:13:01.155828 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":648,"mutex_wait_us":75,"thread_start_us":96,"threads_started":1}
I20260928 21:13:01.156122 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.150282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46290 (local address 127.0.98.1:45911)
0928 21:13:01.150585 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.150601 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.150838 (+ 237us) server_negotiation.cc:407] Connection header received
0928 21:13:01.151053 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.151057 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.151256 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.151401 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:01.151815 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.152928 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.154424 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.155163 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.155259 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.155388 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.155887 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.155987 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":71,"threads_started":1}
I20260928 21:13:01.156594 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.149847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45911 (local address 127.0.0.1:46290)
0928 21:13:01.150733 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.150745 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:01.150901 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:01.151553 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:01.151559 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:01.151581 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:01.151685 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.151691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.153052 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.153055 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.154265 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.154275 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.155663 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.155672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.155736 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.155754 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:13:01.155855 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:13:01.155928 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"mutex_wait_us":33,"thread_start_us":73,"threads_started":1}
I20260928 21:13:01.156728 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.150163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.0.98.2:38305)
0928 21:13:01.150767 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.150780 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.150804 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:13:01.150890 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.150896 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.151107 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.151268 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:01.151563 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.152763 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.154404 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.155236 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.155348 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.156372 (+ 1024us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.156422 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.156510 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":108,"threads_started":1}
I20260928 21:13:01.158344 11044 tablet_service.cc:1467] Processing CreateTablet for tablet 93139575f79b4227bd1b24ce68960e46 (DEFAULT_TABLE table=test-table [id=5f8d3721c01c4ee781db991dad1dd069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:13:01.159298 11044 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93139575f79b4227bd1b24ce68960e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:01.159302 10776 tablet_service.cc:1467] Processing CreateTablet for tablet 93139575f79b4227bd1b24ce68960e46 (DEFAULT_TABLE table=test-table [id=5f8d3721c01c4ee781db991dad1dd069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:13:01.159759 10910 tablet_service.cc:1467] Processing CreateTablet for tablet 93139575f79b4227bd1b24ce68960e46 (DEFAULT_TABLE table=test-table [id=5f8d3721c01c4ee781db991dad1dd069]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:13:01.160329 10776 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93139575f79b4227bd1b24ce68960e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:01.160671 10910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 93139575f79b4227bd1b24ce68960e46. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:01.165666 11127 tablet_bootstrap.cc:492] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Bootstrap starting.
I20260928 21:13:01.167996 11128 tablet_bootstrap.cc:492] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Bootstrap starting.
I20260928 21:13:01.168210 11127 tablet_bootstrap.cc:654] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:01.169317 11127 log.cc:824] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:01.169900 11129 tablet_bootstrap.cc:492] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093: Bootstrap starting.
I20260928 21:13:01.170275 11128 tablet_bootstrap.cc:654] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:01.171056 11128 log.cc:824] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:01.171417 11127 tablet_bootstrap.cc:492] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: No bootstrap required, opened a new log
I20260928 21:13:01.171619 11127 ts_tablet_manager.cc:1397] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:13:01.171630 11129 tablet_bootstrap.cc:654] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:01.172924 11129 log.cc:824] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:01.172968 11128 tablet_bootstrap.cc:492] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: No bootstrap required, opened a new log
I20260928 21:13:01.173141 11128 ts_tablet_manager.cc:1397] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:13:01.174561 11129 tablet_bootstrap.cc:492] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093: No bootstrap required, opened a new log
I20260928 21:13:01.174742 11129 ts_tablet_manager.cc:1397] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:13:01.175551 11127 raft_consensus.cc:348] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.175835 11127 raft_consensus.cc:374] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:01.175894 11127 raft_consensus.cc:729] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 359617309f0b407fa109909806fddf3a, State: Initialized, Role: FOLLOWER
I20260928 21:13:01.176347 11127 consensus_queue.cc:260] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.176847 11128 raft_consensus.cc:348] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.177066 11110 heartbeater.cc:503] Master 127.0.98.62:44749 was elected leader, sending a full tablet report...
I20260928 21:13:01.177091 11128 raft_consensus.cc:374] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:01.177145 11128 raft_consensus.cc:729] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b61713c5a7f347b3a5a7ccf3c116261e, State: Initialized, Role: FOLLOWER
I20260928 21:13:01.177564 11127 ts_tablet_manager.cc:1428] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:13:01.177573 11128 consensus_queue.cc:260] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.178211 10976 heartbeater.cc:503] Master 127.0.98.62:44749 was elected leader, sending a full tablet report...
I20260928 21:13:01.178059 11129 raft_consensus.cc:348] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.178301 11129 raft_consensus.cc:374] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:01.178364 11129 raft_consensus.cc:729] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9ac4436e524db7a5868d9a056b3093, State: Initialized, Role: FOLLOWER
I20260928 21:13:01.178807 11129 consensus_queue.cc:260] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.179785 11129 ts_tablet_manager.cc:1428] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:13:01.180083 10842 heartbeater.cc:503] Master 127.0.98.62:44749 was elected leader, sending a full tablet report...
I20260928 21:13:01.180683 11128 ts_tablet_manager.cc:1428] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:13:01.213097 11133 raft_consensus.cc:482] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:13:01.213370 11133 raft_consensus.cc:504] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.214637 11133 leader_election.cc:288] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b61713c5a7f347b3a5a7ccf3c116261e (127.0.98.2:38305), cf9ac4436e524db7a5868d9a056b3093 (127.0.98.1:45911)
I20260928 21:13:01.218451 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.215037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42371 (local address 127.0.98.1:45911)
0928 21:13:01.215157 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.215162 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.215353 (+ 191us) server_negotiation.cc:407] Connection header received
0928 21:13:01.215489 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.215497 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.215540 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.215627 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:01.215894 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.216377 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.217506 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.217879 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.218005 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.218068 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.218252 (+ 184us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.218321 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:13:01.218454 11136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.214899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45911 (local address 127.0.98.3:42371)
0928 21:13:01.215248 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.215259 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:01.215386 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:01.215638 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:01.215641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:01.215656 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:01.215779 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.215784 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.216520 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.216523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.217347 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.217356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.218010 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.218013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.218109 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.218132 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:13:01.218226 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:13:01.218302 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":117,"threads_started":1}
I20260928 21:13:01.218649 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.214956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60611 (local address 127.0.98.2:38305)
0928 21:13:01.215089 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.215094 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.215112 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:13:01.215300 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.215303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.215350 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.215456 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:01.215745 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.216291 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.217592 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.218028 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.218157 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.218225 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.218427 (+ 202us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.218498 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:13:01.218657 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.214847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38305 (local address 127.0.98.3:60611)
0928 21:13:01.215047 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.215065 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:01.215164 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:01.215459 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:01.215462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:01.215481 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:01.215619 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.215631 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.216413 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.216420 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.217432 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:01.217442 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.218154 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.218157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:01.218284 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.218305 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:13:01.218409 (+ 104us) client_negotiation.cc:240] Negotiation successful
0928 21:13:01.218489 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"mutex_wait_us":1}
I20260928 21:13:01.219133 10796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93139575f79b4227bd1b24ce68960e46" candidate_uuid: "359617309f0b407fa109909806fddf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9ac4436e524db7a5868d9a056b3093" is_pre_election: true
I20260928 21:13:01.219338 10930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93139575f79b4227bd1b24ce68960e46" candidate_uuid: "359617309f0b407fa109909806fddf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" is_pre_election: true
I20260928 21:13:01.219486 10796 raft_consensus.cc:2454] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359617309f0b407fa109909806fddf3a in term 0.
I20260928 21:13:01.219595 10930 raft_consensus.cc:2454] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 359617309f0b407fa109909806fddf3a in term 0.
I20260928 21:13:01.220041 10998 leader_election.cc:302] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 359617309f0b407fa109909806fddf3a, b61713c5a7f347b3a5a7ccf3c116261e, cf9ac4436e524db7a5868d9a056b3093; no voters:
I20260928 21:13:01.220369 11133 raft_consensus.cc:2781] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:13:01.220460 11133 raft_consensus.cc:482] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:13:01.220516 11133 raft_consensus.cc:3037] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:01.221671 11133 raft_consensus.cc:504] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.222028 11133 leader_election.cc:288] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [CANDIDATE]: Term 1 election: Requested vote from peers b61713c5a7f347b3a5a7ccf3c116261e (127.0.98.2:38305), cf9ac4436e524db7a5868d9a056b3093 (127.0.98.1:45911)
I20260928 21:13:01.222342 10930 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93139575f79b4227bd1b24ce68960e46" candidate_uuid: "359617309f0b407fa109909806fddf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b61713c5a7f347b3a5a7ccf3c116261e"
I20260928 21:13:01.222404 10796 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "93139575f79b4227bd1b24ce68960e46" candidate_uuid: "359617309f0b407fa109909806fddf3a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9ac4436e524db7a5868d9a056b3093"
I20260928 21:13:01.222493 10930 raft_consensus.cc:3037] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:01.222575 10796 raft_consensus.cc:3037] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:01.223598 10930 raft_consensus.cc:2454] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359617309f0b407fa109909806fddf3a in term 1.
I20260928 21:13:01.223647 10796 raft_consensus.cc:2454] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 359617309f0b407fa109909806fddf3a in term 1.
I20260928 21:13:01.223924 10997 leader_election.cc:302] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 359617309f0b407fa109909806fddf3a, b61713c5a7f347b3a5a7ccf3c116261e; no voters:
I20260928 21:13:01.224120 11133 raft_consensus.cc:2781] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:13:01.224355 11133 raft_consensus.cc:686] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 1 LEADER]: Becoming Leader. State: Replica: 359617309f0b407fa109909806fddf3a, State: Running, Role: LEADER
I20260928 21:13:01.224761 11133 consensus_queue.cc:237] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } }
I20260928 21:13:01.228040 10651 catalog_manager.cc:5495] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a reported cstate change: term changed from 0 to 1, leader changed from <none> to 359617309f0b407fa109909806fddf3a (127.0.98.3). New cstate: current_term: 1 leader_uuid: "359617309f0b407fa109909806fddf3a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "359617309f0b407fa109909806fddf3a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46741 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 } health_report { overall_health: UNKNOWN } } }
I20260928 21:13:01.255195 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.251648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55792 (local address 127.0.98.3:46741)
0928 21:13:01.251785 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:01.251789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:01.251803 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:13:01.251961 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:01.251964 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:01.252019 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:01.252112 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:13:01.252346 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.252858 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.253728 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:01.253956 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:01.254044 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:01.254384 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:13:01.254782 (+ 398us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:13:01.254895 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:01.255015 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:13:01.255068 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:13:01.261377 10930 raft_consensus.cc:1261] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 1 FOLLOWER]: Refusing update from remote peer 359617309f0b407fa109909806fddf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:13:01.261395 10796 raft_consensus.cc:1261] T 93139575f79b4227bd1b24ce68960e46 P cf9ac4436e524db7a5868d9a056b3093 [term 1 FOLLOWER]: Refusing update from remote peer 359617309f0b407fa109909806fddf3a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:13:01.262302 11133 consensus_queue.cc:1035] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "b61713c5a7f347b3a5a7ccf3c116261e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 38305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:13:01.262534 11137 consensus_queue.cc:1035] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9ac4436e524db7a5868d9a056b3093" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:13:01.274631 11141 mvcc.cc:205] Tried to move back new op lower bound from 7334420403238240256 to 7334420403099287552. Current Snapshot: MvccSnapshot[applied={T|T < 7334420403238240256}]
I20260928 21:13:01.279696 10651 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:13:01.279920 10651 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54326:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:13:01.282516 10651 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 552d4b72c5d8431aa9a86f1c0d811e54: Sending DeleteTablet for 3 replicas of tablet 93139575f79b4227bd1b24ce68960e46
I20260928 21:13:01.283648 11044 tablet_service.cc:1514] Processing DeleteTablet for tablet 93139575f79b4227bd1b24ce68960e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55784
I20260928 21:13:01.283898 10910 tablet_service.cc:1514] Processing DeleteTablet for tablet 93139575f79b4227bd1b24ce68960e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53714
I20260928 21:13:01.284303 11149 tablet_replica.cc:330] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: stopping tablet replica
I20260928 21:13:01.284401 11150 tablet_replica.cc:330] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: stopping tablet replica
I20260928 21:13:01.284755 11150 raft_consensus.cc:2229] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:13:01.284755 11149 raft_consensus.cc:2229] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:13:01.284751 10776 tablet_service.cc:1514] Processing DeleteTablet for tablet 93139575f79b4227bd1b24ce68960e46 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:13:01 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46290
I20260928 21:13:01.285157 11150 raft_consensus.cc:2258] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:13:01.285383 11149 raft_consensus.cc:2258] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:13:01.286242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10713
I20260928 21:13:01.287609 11149 ts_tablet_manager.cc:1905] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:13:01.289958 11150 ts_tablet_manager.cc:1905] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:13:01.290987 11149 ts_tablet_manager.cc:1918] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:13:01.291164 11149 log.cc:1196] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/93139575f79b4227bd1b24ce68960e46
I20260928 21:13:01.291709 11149 ts_tablet_manager.cc:1939] T 93139575f79b4227bd1b24ce68960e46 P 359617309f0b407fa109909806fddf3a: Deleting consensus metadata
I20260928 21:13:01.293102 10637 catalog_manager.cc:4841] TS 359617309f0b407fa109909806fddf3a (127.0.98.3:46741): tablet 93139575f79b4227bd1b24ce68960e46 (table test-table [id=5f8d3721c01c4ee781db991dad1dd069]) successfully deleted
I20260928 21:13:01.293896 11150 ts_tablet_manager.cc:1918] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:13:01.294013 11150 log.cc:1196] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/93139575f79b4227bd1b24ce68960e46
I20260928 21:13:01.294407 11150 ts_tablet_manager.cc:1939] T 93139575f79b4227bd1b24ce68960e46 P b61713c5a7f347b3a5a7ccf3c116261e: Deleting consensus metadata
W20260928 21:13:01.294508 10636 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45911 (error 108)
I20260928 21:13:01.294731 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10847
I20260928 21:13:01.296192 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.295889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45911 (local address 127.0.0.1:46298)
0928 21:13:01.296041 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.296115 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 21:13:01.296345 10636 catalog_manager.cc:4584] TS cf9ac4436e524db7a5868d9a056b3093 (127.0.98.1:45911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 93139575f79b4227bd1b24ce68960e46: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45911: connect: Connection refused (error 111)
I20260928 21:13:01.302809 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10981
I20260928 21:13:01.303020 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:01.302728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:38305 (local address 127.0.0.1:53726)
0928 21:13:01.302876 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:01.302935 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:38305: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:13:01.309273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 10619
2026-09-28T21:13:01Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2321 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:13:01 dist-test-slave-v7xz krb5kdc[11155](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:13:01 dist-test-slave-v7xz krb5kdc[11155](info): set up 2 sockets
Sep 28 21:13:01 dist-test-slave-v7xz krb5kdc[11155](info): commencing operation
krb5kdc: starting...
W20260928 21:13:02.349750 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:13:02 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790629982, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:13:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:13:02Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 21:13:02.516687 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8iL2mC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/postgres -l logfile start
2026-09-28 21:13:05.024 UTC [11185] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:13:05.024 UTC [11185] LOG: listening on IPv4 address "127.0.98.20", port 41615
2026-09-28 21:13:05.027 UTC [11185] LOG: listening on Unix socket "/tmp/.s.PGSQL.41615"
2026-09-28 21:13:05.061 UTC [11188] LOG: database system was shut down at 2026-09-28 21:13:03 UTC
2026-09-28 21:13:05.065 UTC [11185] LOG: database system is ready to accept connections
I20260928 21:13:06.009512 392 mini_postgres.cc:96] Postgres bound to 41615
2026-09-28 21:13:06.015 UTC [11196] FATAL: database "slave" does not exist
127.0.98.20:41615 - accepting connections
I20260928 21:13:06.016526 392 mini_ranger.cc:161] Starting Ranger...
I20260928 21:13:06.032490 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:13:06.509712 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:13:06.509814 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-admin
2026-09-28 21:13:06,787 [I] DB FLAVOR :POSTGRES
2026-09-28 21:13:06,788 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:13:06,788 [I] Checking connection..
2026-09-28 21:13:06,788 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:13:07,031 [I] Checking connection passed.
2026-09-28 21:13:07,031 [I] --------- Verifying version history table ---------
2026-09-28 21:13:07,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:13:07,292 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:13:07,292 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:13:07,547 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:13:07,547 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:13:07,547 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:13:07.777 UTC [11260] WARNING: there is no transaction in progress
2026-09-28 21:13:07,783 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:13:07,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:13:08,015 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:13:08,015 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:13:08,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:13:08,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:13:08,504 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v7xz.c.gcp-upstream.internal','N') ;"
2026-09-28 21:13:08,728 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:13:08,980 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:13:08,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:13:09,249 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:13:09,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:13:09,477 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:13:09,732 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:13:09,733 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:13:10.741 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.757 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.774 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.790 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.808 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.825 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.881 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.888 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.903 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.913 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.924 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.938 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.947 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.954 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.961 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:10.968 UTC [11368] WARNING: there is no transaction in progress
2026-09-28 21:13:11,655 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:13:11,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:13:11,912 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:13:11,912 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:13:12,164 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:13:12,165 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:13:12,408 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:13:12,408 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:13:12,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:13:12,939 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v7xz.c.gcp-upstream.internal';"
2026-09-28 21:13:13,192 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:13:13,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:13:13,435 [I] DB_PATCHES have already been applied
I20260928 21:13:13.440686 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8iL2mC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:13:13.440745 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 21:13:13.444106 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:33067
Sep 28, 2026 9:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:13:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:13:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:13:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:13:14.561371 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.545s user 0.000s sys 0.010s
Sep 28, 2026 9:13:35 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:13:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 28, 2026 9:13:35 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:13:35 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:13:36 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:13:36.637912 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:13:36 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:13:36.722512 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:13:36.732010 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:13:37.570842 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:13:37.570925 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms
I20260928 21:13:37.570979 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41615
2026-09-28 21:13:37,755 [I] DB FLAVOR :POSTGRES
2026-09-28 21:13:37,755 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:13:37,755 [I] Checking connection
2026-09-28 21:13:37,755 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:13:38,015 [I] connection success
2026-09-28 21:13:38,015 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:13:38,015 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:13:38,015 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:13:38,260 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:13:38,261 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:13:38,261 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:13:38,261 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41615/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:13:38,509 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:13:38.957151 392 mini_ranger_kms.cc:310] Created kms service
I20260928 21:13:39.164304 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:13:39.207335 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:13:39 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630016, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:39.624004 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:13:39.624161 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:13:39.624186 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 21:13:39.628305 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11548
I20260928 21:13:39.628537 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:32937
Sep 28, 2026 9:13:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:13:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:13:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:13:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:13:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:13:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:13:41.897063 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.175s user 0.002s sys 0.166s
I20260928 21:13:41.897203 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:13:41.897351 392 mini_ranger_kms.cc:401] 127.0.98.20:32937/kms/v1/keys
Sep 28 21:13:42 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630022, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:13:42 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630022, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 21:13:42 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630022, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 21:13:45 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:13:45.141458 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46279
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46279
--ranger_config_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:32937/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:45.258523 11599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:45.258788 11599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:45.258829 11599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:45.262889 11599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:13:45.262950 11599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:45.262969 11599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:45.262986 11599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:13:45.263001 11599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:13:45.268532 11599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:32937/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46279
--ranger_config_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:45.269610 11599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:45.270632 11599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:45.280344 11607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:45.280380 11605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:45.280475 11599 server_base.cc:1034] running on GCE node
W20260928 21:13:45.280663 11604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:45.281262 11599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:45.282380 11599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:45.283552 11599 hybrid_clock.cc:648] HybridClock initialized: now 1790630025283526 us; error 44 us; skew 500 ppm
Sep 28 21:13:45 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:45.290239 11599 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:13:45.291246 11599 webserver.cc:460] Webserver started at http://127.0.98.62:35993/ using document root <none> and password file <none>
I20260928 21:13:45.291808 11599 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:45.291888 11599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:45.292107 11599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:13:45 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:45.350301 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be"
format_stamp: "Formatted at 2026-09-28 21:13:45 on dist-test-slave-v7xz"
server_key: "f208f17dddeff6e4f9892e04511ff3f3"
server_key_iv: "5be37861d5fa401843079c41892d9666"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:45.350981 11599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be"
format_stamp: "Formatted at 2026-09-28 21:13:45 on dist-test-slave-v7xz"
server_key: "f208f17dddeff6e4f9892e04511ff3f3"
server_key_iv: "5be37861d5fa401843079c41892d9666"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:45.355754 11599 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:13:45.371402 11614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:45.373063 11599 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:13:45.373226 11599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be"
format_stamp: "Formatted at 2026-09-28 21:13:45 on dist-test-slave-v7xz"
server_key: "f208f17dddeff6e4f9892e04511ff3f3"
server_key_iv: "5be37861d5fa401843079c41892d9666"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:45.373377 11599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:45.388590 11599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:45.391695 11599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:45.391918 11599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:45.400818 11599 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46279
I20260928 21:13:45.400893 11666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46279 every 8 connection(s)
I20260928 21:13:45.401873 11599 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:13:45.405467 11667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:45.409664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 11599
I20260928 21:13:45.409816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:13:45.410115 392 external_mini_cluster.cc:1428] Setting key d822db57f7c5dcced3a3042e7b35d9d9
I20260928 21:13:45.412776 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be: Bootstrap starting.
I20260928 21:13:45.415555 11667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:45.416421 11667 log.cc:824] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:45.418659 11667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be: No bootstrap required, opened a new log
Sep 28 21:13:45 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630025, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:13:45.423141 11667 raft_consensus.cc:348] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46279 } }
I20260928 21:13:45.423408 11667 raft_consensus.cc:374] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:45.423492 11667 raft_consensus.cc:729] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf8ecbf8289e4c6eb36a5b95ac78c4be, State: Initialized, Role: FOLLOWER
I20260928 21:13:45.423995 11667 consensus_queue.cc:260] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46279 } }
I20260928 21:13:45.424098 11670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:45.411593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39952 (local address 127.0.98.62:46279)
0928 21:13:45.412005 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:45.412031 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:45.412068 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:13:45.412670 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:45.412687 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:45.413115 (+ 428us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:45.413467 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:45.414262 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:45.415224 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:45.416171 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:45.416704 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:45.416899 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:45.419485 (+ 2586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:45.419491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:45.419513 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:45.419539 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:45.421261 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:45.421702 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:45.421712 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:45.421716 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:45.421818 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:45.422099 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:45.422103 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:45.422104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:45.422483 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:45.422693 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:45.422950 (+ 257us) server_negotiation.cc:299] Negotiation successful
0928 21:13:45.423238 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":145,"threads_started":1}
I20260928 21:13:45.424160 11667 raft_consensus.cc:388] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:13:45.424371 11667 raft_consensus.cc:482] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:13:45.424502 11667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:45.425645 11667 raft_consensus.cc:504] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46279 } }
I20260928 21:13:45.426019 11667 leader_election.cc:302] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cf8ecbf8289e4c6eb36a5b95ac78c4be; no voters:
I20260928 21:13:45.426384 11667 leader_election.cc:288] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:13:45.426479 11672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:13:45.426728 11672 raft_consensus.cc:686] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [term 1 LEADER]: Becoming Leader. State: Replica: cf8ecbf8289e4c6eb36a5b95ac78c4be, State: Running, Role: LEADER
I20260928 21:13:45.427085 11672 consensus_queue.cc:237] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46279 } }
I20260928 21:13:45.427928 11667 sys_catalog.cc:564] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:13:45.428139 11672 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46279 } } }
I20260928 21:13:45.428354 11672 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [sys.catalog]: This master's current role is: LEADER
I20260928 21:13:45.428537 11673 sys_catalog.cc:455] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [sys.catalog]: SysCatalogTable state changed. Reason: New leader cf8ecbf8289e4c6eb36a5b95ac78c4be. Latest consensus state: current_term: 1 leader_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cf8ecbf8289e4c6eb36a5b95ac78c4be" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46279 } } }
I20260928 21:13:45.428620 11673 sys_catalog.cc:458] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be [sys.catalog]: This master's current role is: LEADER
I20260928 21:13:45.428781 11676 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:13:45.431764 11676 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:13:45.432195 11667 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:13:45.441571 11676 catalog_manager.cc:1347] Generated new cluster ID: 1774bcac1b244f1eb6bed34cf8884fb1
I20260928 21:13:45.441648 11676 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:13:45.454749 11676 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:13:45.456126 11676 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:13:45.469344 11676 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be: Generated new TSK 0
I20260928 21:13:45.470242 11676 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:13:46.123857 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:32937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46279
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:46.264932 11712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:46.265434 11712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:46.265555 11712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:46.272562 11712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:46.272651 11712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:46.272753 11712 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:13:46.278632 11712 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:32937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11712
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:46.280236 11712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:46.281582 11712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:46.292887 11719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:46.292887 11720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:46.293320 11722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:46.293459 11712 server_base.cc:1034] running on GCE node
I20260928 21:13:46.293890 11712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:46.294612 11712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:46.295861 11712 hybrid_clock.cc:648] HybridClock initialized: now 1790630026295827 us; error 53 us; skew 500 ppm
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:46.298660 11712 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:13:46.299642 11712 webserver.cc:460] Webserver started at http://127.0.98.1:44573/ using document root <none> and password file <none>
I20260928 21:13:46.300215 11712 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:46.300307 11712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:46.300545 11712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:46.315506 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "ababe079d1d04afd829bd891cc513e2d"
format_stamp: "Formatted at 2026-09-28 21:13:46 on dist-test-slave-v7xz"
server_key: "aca71f6c16747dcba1fce87b397e2a8b"
server_key_iv: "f459b51acb7c264e97a155c4718d9d1c"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:46.316447 11712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "ababe079d1d04afd829bd891cc513e2d"
format_stamp: "Formatted at 2026-09-28 21:13:46 on dist-test-slave-v7xz"
server_key: "aca71f6c16747dcba1fce87b397e2a8b"
server_key_iv: "f459b51acb7c264e97a155c4718d9d1c"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:46.321599 11712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260928 21:13:46.352280 11729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:46.354158 11712 fs_manager.cc:725] Time spent opening block manager: real 0.019s user 0.004s sys 0.000s
I20260928 21:13:46.354316 11712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "ababe079d1d04afd829bd891cc513e2d"
format_stamp: "Formatted at 2026-09-28 21:13:46 on dist-test-slave-v7xz"
server_key: "aca71f6c16747dcba1fce87b397e2a8b"
server_key_iv: "f459b51acb7c264e97a155c4718d9d1c"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:46.354470 11712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:46.386482 11712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:46.390432 11712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:46.390692 11712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:46.391409 11712 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:13:46.392778 11712 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:13:46.392904 11712 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:46.393033 11712 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:13:46.393118 11712 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:46.404904 11712 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41593
I20260928 21:13:46.405050 11844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41593 every 8 connection(s)
I20260928 21:13:46.406176 11712 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:13:46.413329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 11712
I20260928 21:13:46.413465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:13:46.413810 392 external_mini_cluster.cc:1428] Setting key 868d35463c5e57e18bd6c251135400a1
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:13:46.431700 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:46.410780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49391 (local address 127.0.98.62:46279)
0928 21:13:46.416298 (+ 5518us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:46.416308 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:46.416326 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:13:46.417140 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:46.417147 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:46.417214 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:46.417300 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:46.418929 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.419755 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.421529 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.421904 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.422003 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:46.426174 (+ 4171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:46.426180 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:46.426183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:46.426215 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:46.428374 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:46.429591 (+ 1217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:46.429595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:46.429597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:46.429689 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:46.430160 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:46.430165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:46.430168 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:46.430442 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:46.430539 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:46.431392 (+ 853us) server_negotiation.cc:299] Negotiation successful
0928 21:13:46.431516 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5434,"thread_start_us":104,"threads_started":1}
I20260928 21:13:46.432724 11847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:46.409603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46279 (local address 127.0.98.1:49391)
0928 21:13:46.413989 (+ 4386us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:46.414057 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:46.416729 (+ 2672us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:46.417627 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:46.417637 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:46.418036 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:13:46.418747 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:46.418767 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.419916 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.419919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:46.421385 (+ 1466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:46.421397 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.422179 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.422183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:46.422398 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:46.424079 (+ 1681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:13:46.424101 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:13:46.425946 (+ 1845us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:13:46.428539 (+ 2593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:46.428544 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:46.428562 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:46.429464 (+ 902us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:46.429836 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:46.429839 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:46.429842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:46.430041 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:46.430595 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:13:46.430602 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:13:46.430954 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 21:13:46.431182 (+ 228us) client_negotiation.cc:240] Negotiation successful
0928 21:13:46.431569 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4116,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:13:46.434487 11845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46279
I20260928 21:13:46.434728 11845 heartbeater.cc:464] Registering TS with master...
I20260928 21:13:46.435310 11845 heartbeater.cc:511] Master 127.0.98.62:46279 requested a full tablet report, sending...
I20260928 21:13:46.437132 11631 ts_manager.cc:194] Registered new tserver with Master: ababe079d1d04afd829bd891cc513e2d (127.0.98.1:41593)
I20260928 21:13:46.438777 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:49391
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:13:46.452144 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:32937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46279
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:13:46.585292 11851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:46.585675 11851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:46.585781 11851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:46.590624 11851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:46.590790 11851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:46.590935 11851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:13:46.596406 11851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:32937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:46.597553 11851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:46.598569 11851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:46.610221 11858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:46.610569 11851 server_base.cc:1034] running on GCE node
W20260928 21:13:46.610245 11857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:46.610255 11860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:46.611266 11851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:46.612073 11851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:46.613279 11851 hybrid_clock.cc:648] HybridClock initialized: now 1790630026613240 us; error 50 us; skew 500 ppm
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:46.617043 11851 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:13:46.618144 11851 webserver.cc:460] Webserver started at http://127.0.98.2:40457/ using document root <none> and password file <none>
I20260928 21:13:46.618849 11851 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:46.618934 11851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:46.619155 11851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:46.639394 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "d99d3c4456414a028a3f018ca7a04382"
format_stamp: "Formatted at 2026-09-28 21:13:46 on dist-test-slave-v7xz"
server_key: "947fe4a1e8c320c306612cf86ca4dc54"
server_key_iv: "482051f48ff300287744af2940e84188"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:46.640277 11851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d99d3c4456414a028a3f018ca7a04382"
format_stamp: "Formatted at 2026-09-28 21:13:46 on dist-test-slave-v7xz"
server_key: "947fe4a1e8c320c306612cf86ca4dc54"
server_key_iv: "482051f48ff300287744af2940e84188"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:46.646322 11851 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260928 21:13:46.673348 11867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:46.675233 11851 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.003s sys 0.000s
I20260928 21:13:46.675364 11851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "d99d3c4456414a028a3f018ca7a04382"
format_stamp: "Formatted at 2026-09-28 21:13:46 on dist-test-slave-v7xz"
server_key: "947fe4a1e8c320c306612cf86ca4dc54"
server_key_iv: "482051f48ff300287744af2940e84188"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:46.675498 11851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:46.731041 11851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:46.736145 11851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:46.736443 11851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:46.737236 11851 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:13:46.738358 11851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:13:46.738466 11851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:46.738571 11851 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:13:46.738642 11851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:46.750089 11851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40663
I20260928 21:13:46.751385 11851 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:13:46.754341 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 11851
I20260928 21:13:46.754491 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:13:46.754812 392 external_mini_cluster.cc:1428] Setting key be55ce8bc2e90ae92c4b06d2468ef67e
I20260928 21:13:46.765033 11980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40663 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:13:46.786445 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:46.768683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37423 (local address 127.0.98.62:46279)
0928 21:13:46.768872 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:46.768878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:46.772413 (+ 3535us) server_negotiation.cc:407] Connection header received
0928 21:13:46.773527 (+ 1114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:46.773531 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:46.773591 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:46.773761 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:46.775388 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.776261 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.777984 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.778406 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.778530 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:46.782010 (+ 3480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:46.782014 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:46.782019 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:46.782054 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:46.784025 (+ 1971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:46.784682 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:46.784688 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:46.784690 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:46.784780 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:46.785206 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:46.785210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:46.785212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:46.785429 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:46.785536 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:46.786140 (+ 604us) server_negotiation.cc:299] Negotiation successful
0928 21:13:46.786269 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:13:46.787441 11984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:46.767507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46279 (local address 127.0.98.2:37423)
0928 21:13:46.772225 (+ 4718us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:46.772278 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:46.773292 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:46.774073 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:46.774082 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:46.774464 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:13:46.775183 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:46.775207 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.776465 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.776470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:46.777846 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:46.777855 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:46.778760 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:46.778767 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:46.779020 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:46.779779 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:13:46.779807 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:13:46.781782 (+ 1975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:13:46.784175 (+ 2393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:46.784181 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:46.784198 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:46.784558 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:46.784957 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:46.784959 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:46.784960 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:46.785093 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:46.785536 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:13:46.785544 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:13:46.785837 (+ 293us) client_negotiation.cc:769] Sending connection context
0928 21:13:46.786035 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 21:13:46.786368 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4434,"thread_start_us":175,"threads_started":1}
I20260928 21:13:46.788650 11981 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46279
I20260928 21:13:46.788954 11981 heartbeater.cc:464] Registering TS with master...
I20260928 21:13:46.789487 11981 heartbeater.cc:511] Master 127.0.98.62:46279 requested a full tablet report, sending...
I20260928 21:13:46.790578 11631 ts_manager.cc:194] Registered new tserver with Master: d99d3c4456414a028a3f018ca7a04382 (127.0.98.2:40663)
I20260928 21:13:46.791210 11631 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:13:46.793231 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:32937/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46279
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43145
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:13:46.955504 11986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:13:46.956346 11986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:13:46.956497 11986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:13:46.962636 11986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:13:46.962822 11986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:13:46.962965 11986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:13:46.971028 11986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43145
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:32937/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11986
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:13:46.972700 11986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:13:46.974143 11986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:13:46.987233 11992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:13:46.987778 11991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:46.988574 11986 server_base.cc:1034] running on GCE node
W20260928 21:13:46.988874 11994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:13:46.989390 11986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:13:46.990149 11986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:13:46.991305 11986 hybrid_clock.cc:648] HybridClock initialized: now 1790630026991240 us; error 81 us; skew 500 ppm
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:13:46.994946 11986 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:13:46.996673 11986 webserver.cc:460] Webserver started at http://127.0.98.3:41827/ using document root <none> and password file <none>
I20260928 21:13:46.997481 11986 fs_manager.cc:359] Metadata directory not provided
I20260928 21:13:46.997627 11986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:13:46.998890 11986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:13:46 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:47.021304 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "640ca2bbb6074fb3ac108251f27e0742"
format_stamp: "Formatted at 2026-09-28 21:13:47 on dist-test-slave-v7xz"
server_key: "9fe5442e78cac333f37d579adffe5839"
server_key_iv: "0ba6c09fe5ddc553eceb7ab04c824a73"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:47.029942 11986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "640ca2bbb6074fb3ac108251f27e0742"
format_stamp: "Formatted at 2026-09-28 21:13:47 on dist-test-slave-v7xz"
server_key: "9fe5442e78cac333f37d579adffe5839"
server_key_iv: "0ba6c09fe5ddc553eceb7ab04c824a73"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:47.040263 11986 fs_manager.cc:691] Time spent creating directory manager: real 0.010s user 0.005s sys 0.000s
Sep 28 21:13:47 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:47.063107 12001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:47.064880 11986 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260928 21:13:47.065007 11986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "640ca2bbb6074fb3ac108251f27e0742"
format_stamp: "Formatted at 2026-09-28 21:13:47 on dist-test-slave-v7xz"
server_key: "9fe5442e78cac333f37d579adffe5839"
server_key_iv: "0ba6c09fe5ddc553eceb7ab04c824a73"
server_key_version: "kuduclusterkey@0"
I20260928 21:13:47.065132 11986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:13:47.102723 11986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:13:47.106653 11986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:13:47.106971 11986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:13:47.107712 11986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:13:47.108861 11986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:13:47.108947 11986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:47.109021 11986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:13:47.109063 11986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:13:47.119936 11986 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43407
I20260928 21:13:47.121248 11986 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:13:47.133138 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 11986
I20260928 21:13:47.133267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:13:47.133589 392 external_mini_cluster.cc:1428] Setting key b5cf6e0452e0e919d9577db0f5d47213
I20260928 21:13:47.133656 12114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43407 every 8 connection(s)
Sep 28 21:13:47 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:13:47.170728 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.140690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60341 (local address 127.0.98.62:46279)
0928 21:13:47.140805 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.140810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.145341 (+ 4531us) server_negotiation.cc:407] Connection header received
0928 21:13:47.146608 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.146612 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.146666 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.146718 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:47.152926 (+ 6208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.153742 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.155475 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.155799 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.155872 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.166324 (+ 10452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:47.166328 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:47.166331 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:47.166360 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:47.168120 (+ 1760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:47.168942 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:47.168945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:47.168947 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:47.169038 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:47.169448 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:47.169451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:47.169453 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:47.169665 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:47.169737 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.170439 (+ 702us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.170574 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:13:47.172613 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.139556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46279 (local address 127.0.98.3:60341)
0928 21:13:47.145161 (+ 5605us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.145228 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:47.146352 (+ 1124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:47.147184 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:47.147194 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:47.147565 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:13:47.148254 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.148274 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.153923 (+ 5649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.153926 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.155338 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.155350 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.156073 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.156077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.156278 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.164233 (+ 7955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:13:47.164258 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:13:47.166113 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:13:47.168408 (+ 2295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:47.168415 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:47.168435 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:47.168804 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:47.169145 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:13:47.169149 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:13:47.169152 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:13:47.169345 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:13:47.169811 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:13:47.169817 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:13:47.170167 (+ 350us) client_negotiation.cc:769] Sending connection context
0928 21:13:47.171178 (+ 1011us) client_negotiation.cc:240] Negotiation successful
0928 21:13:47.171546 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5349,"thread_start_us":189,"threads_started":1}
I20260928 21:13:47.174213 12115 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46279
I20260928 21:13:47.174465 12115 heartbeater.cc:464] Registering TS with master...
I20260928 21:13:47.175048 12115 heartbeater.cc:511] Master 127.0.98.62:46279 requested a full tablet report, sending...
I20260928 21:13:47.176152 11630 ts_manager.cc:194] Registered new tserver with Master: 640ca2bbb6074fb3ac108251f27e0742 (127.0.98.3:43407)
I20260928 21:13:47.176944 11630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60341
I20260928 21:13:47.188123 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:13:47.201107 11848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.191117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39958 (local address 127.0.98.62:46279)
0928 21:13:47.191244 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.191248 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.191718 (+ 470us) server_negotiation.cc:407] Connection header received
0928 21:13:47.191910 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.191913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.191975 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.192031 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:13:47.193060 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.193889 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.195121 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.195471 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.195543 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.197468 (+ 1925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:13:47.197472 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:13:47.197475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:13:47.197501 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:13:47.199192 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:47.199712 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:47.199716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:47.199718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:47.199805 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:13:47.200118 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:13:47.200122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:13:47.200123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:13:47.200322 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:13:47.200443 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.200801 (+ 358us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.200944 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:13:47.205206 11630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39958:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:13:47.209911 11630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:13:47.232901 12127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.219609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40663 (local address 127.0.0.1:44536)
0928 21:13:47.223742 (+ 4133us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.224624 (+ 882us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:47.224714 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:47.225937 (+ 1223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:47.225943 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:47.225969 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:47.226116 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.226123 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.229028 (+ 2905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.229030 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.230997 (+ 1967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.231007 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.232373 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.232375 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.232471 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.232508 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:13:47.232618 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:13:47.232727 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3943,"thread_start_us":111,"threads_started":1}
I20260928 21:13:47.232908 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.220543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44536 (local address 127.0.98.2:40663)
0928 21:13:47.224567 (+ 4024us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.224582 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.225229 (+ 647us) server_negotiation.cc:407] Connection header received
0928 21:13:47.225331 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.225338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.225589 (+ 251us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.225750 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:47.226259 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.228563 (+ 2304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.231187 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.232026 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.232127 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.232272 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.232647 (+ 375us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.232742 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3903,"thread_start_us":61,"threads_started":1}
I20260928 21:13:47.235144 12128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.219697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.0.1:38270)
0928 21:13:47.224039 (+ 4342us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.224105 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:47.224243 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:47.228795 (+ 4552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:47.228797 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:47.228813 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:47.228926 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.228932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.231136 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.231141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.232230 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.232239 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.234447 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.234450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.234542 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.234562 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:13:47.234633 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:13:47.234690 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4271,"thread_start_us":94,"threads_started":1}
I20260928 21:13:47.236353 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8b1db85da0401eac86eae605b71527 (DEFAULT_TABLE table=test-table [id=6ec85d4b1f5e48519bcc650df9b3a86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:13:47.237448 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.220436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.0.98.1:41593)
0928 21:13:47.228065 (+ 7629us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.228073 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.228100 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:13:47.228175 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.228180 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.228467 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.228660 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:47.229761 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.231034 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.233389 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.234146 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.234246 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.234385 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.237203 (+ 2818us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.237312 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7106,"thread_start_us":88,"threads_started":1}
I20260928 21:13:47.237493 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8b1db85da0401eac86eae605b71527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:47.240077 11779 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8b1db85da0401eac86eae605b71527 (DEFAULT_TABLE table=test-table [id=6ec85d4b1f5e48519bcc650df9b3a86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:13:47.240715 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.221320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43407 (local address 127.0.0.1:42650)
0928 21:13:47.228788 (+ 7468us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.228802 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:47.228913 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:47.235637 (+ 6724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:47.235640 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:47.235657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:47.235738 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.235743 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.238210 (+ 2467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.238213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.239270 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.239290 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.240365 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.240372 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.240462 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.240484 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:13:47.240534 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:13:47.240593 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7406,"thread_start_us":102,"threads_started":1}
I20260928 21:13:47.241092 11779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8b1db85da0401eac86eae605b71527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:47.241649 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.221534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42650 (local address 127.0.98.3:43407)
0928 21:13:47.229170 (+ 7636us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.229183 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.232332 (+ 3149us) server_negotiation.cc:407] Connection header received
0928 21:13:47.232412 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.232418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.235494 (+ 3076us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.235877 (+ 383us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:47.236012 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.237061 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.239454 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.240211 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.240981 (+ 770us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.241374 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.241424 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.241499 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7524,"thread_start_us":145,"threads_started":1}
I20260928 21:13:47.244266 12038 tablet_service.cc:1467] Processing CreateTablet for tablet 1a8b1db85da0401eac86eae605b71527 (DEFAULT_TABLE table=test-table [id=6ec85d4b1f5e48519bcc650df9b3a86e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:13:47.245240 12038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a8b1db85da0401eac86eae605b71527. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:13:47.248530 12134 tablet_bootstrap.cc:492] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: Bootstrap starting.
I20260928 21:13:47.251195 12134 tablet_bootstrap.cc:654] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:47.252106 12134 log.cc:824] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:47.253244 12135 tablet_bootstrap.cc:492] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Bootstrap starting.
I20260928 21:13:47.254252 12134 tablet_bootstrap.cc:492] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: No bootstrap required, opened a new log
I20260928 21:13:47.254540 12134 ts_tablet_manager.cc:1397] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:13:47.256500 12135 tablet_bootstrap.cc:654] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:47.257519 12135 log.cc:824] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:47.259482 12134 raft_consensus.cc:348] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.259795 12135 tablet_bootstrap.cc:492] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: No bootstrap required, opened a new log
I20260928 21:13:47.259882 12134 raft_consensus.cc:374] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:47.259934 12134 raft_consensus.cc:729] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ababe079d1d04afd829bd891cc513e2d, State: Initialized, Role: FOLLOWER
I20260928 21:13:47.260010 12135 ts_tablet_manager.cc:1397] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260928 21:13:47.260391 12134 consensus_queue.cc:260] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.261441 12134 ts_tablet_manager.cc:1428] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 21:13:47.261837 11845 heartbeater.cc:503] Master 127.0.98.62:46279 was elected leader, sending a full tablet report...
I20260928 21:13:47.264992 12135 raft_consensus.cc:348] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.265326 12135 raft_consensus.cc:374] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:47.265437 12135 raft_consensus.cc:729] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 640ca2bbb6074fb3ac108251f27e0742, State: Initialized, Role: FOLLOWER
I20260928 21:13:47.265970 12135 consensus_queue.cc:260] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.267091 12135 ts_tablet_manager.cc:1428] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 21:13:47.267270 12115 heartbeater.cc:503] Master 127.0.98.62:46279 was elected leader, sending a full tablet report...
I20260928 21:13:47.268224 12133 tablet_bootstrap.cc:492] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Bootstrap starting.
I20260928 21:13:47.273255 12133 tablet_bootstrap.cc:654] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Neither blocks nor log segments found. Creating new log.
I20260928 21:13:47.274883 12133 log.cc:824] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Log is configured to *not* fsync() on all Append() calls
I20260928 21:13:47.277251 12133 tablet_bootstrap.cc:492] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: No bootstrap required, opened a new log
I20260928 21:13:47.277465 12133 ts_tablet_manager.cc:1397] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260928 21:13:47.281894 12133 raft_consensus.cc:348] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.282262 12133 raft_consensus.cc:374] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:13:47.282353 12133 raft_consensus.cc:729] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d99d3c4456414a028a3f018ca7a04382, State: Initialized, Role: FOLLOWER
I20260928 21:13:47.282966 12133 consensus_queue.cc:260] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.284217 12133 ts_tablet_manager.cc:1428] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Time spent starting tablet: real 0.007s user 0.000s sys 0.006s
I20260928 21:13:47.284644 11981 heartbeater.cc:503] Master 127.0.98.62:46279 was elected leader, sending a full tablet report...
W20260928 21:13:47.385546 12116 tablet.cc:2369] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:13:47.408105 11846 tablet.cc:2369] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 28 21:13:47 dist-test-slave-v7xz krb5kdc[11155](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630026, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:13:47.414397 12141 raft_consensus.cc:482] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:13:47.414577 12141 raft_consensus.cc:504] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.415818 12141 leader_election.cc:288] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ababe079d1d04afd829bd891cc513e2d (127.0.98.1:41593), 640ca2bbb6074fb3ac108251f27e0742 (127.0.98.3:43407)
I20260928 21:13:47.422294 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.416677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46457 (local address 127.0.98.3:43407)
0928 21:13:47.417200 (+ 523us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.417205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.417221 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:13:47.417274 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.417277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.417337 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.417413 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:47.418684 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.419454 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.421013 (+ 1559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.421600 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.421679 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.421745 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.422081 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.422163 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260928 21:13:47.422683 12143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.416046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43407 (local address 127.0.98.2:46457)
0928 21:13:47.416893 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.416909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:47.417027 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:47.418404 (+ 1377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:47.418408 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:47.418430 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:47.418543 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.418550 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.419613 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.419616 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.420810 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.420819 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.421842 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.421845 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.421942 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.421966 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:13:47.422460 (+ 494us) client_negotiation.cc:240] Negotiation successful
0928 21:13:47.422542 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":107,"threads_started":1}
I20260928 21:13:47.423456 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8b1db85da0401eac86eae605b71527" candidate_uuid: "d99d3c4456414a028a3f018ca7a04382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "640ca2bbb6074fb3ac108251f27e0742" is_pre_election: true
I20260928 21:13:47.423800 12050 raft_consensus.cc:2454] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99d3c4456414a028a3f018ca7a04382 in term 0.
I20260928 21:13:47.424389 11869 leader_election.cc:302] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 640ca2bbb6074fb3ac108251f27e0742, d99d3c4456414a028a3f018ca7a04382; no voters:
I20260928 21:13:47.424703 12141 raft_consensus.cc:2781] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:13:47.424808 12141 raft_consensus.cc:482] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:13:47.425815 12141 raft_consensus.cc:3037] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:47.427696 12141 raft_consensus.cc:504] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.428115 12141 leader_election.cc:288] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [CANDIDATE]: Term 1 election: Requested vote from peers ababe079d1d04afd829bd891cc513e2d (127.0.98.1:41593), 640ca2bbb6074fb3ac108251f27e0742 (127.0.98.3:43407)
I20260928 21:13:47.428627 12050 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8b1db85da0401eac86eae605b71527" candidate_uuid: "d99d3c4456414a028a3f018ca7a04382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "640ca2bbb6074fb3ac108251f27e0742"
I20260928 21:13:47.428767 12050 raft_consensus.cc:3037] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:47.430085 12050 raft_consensus.cc:2454] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d99d3c4456414a028a3f018ca7a04382 in term 1.
I20260928 21:13:47.430433 11869 leader_election.cc:302] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 640ca2bbb6074fb3ac108251f27e0742, d99d3c4456414a028a3f018ca7a04382; no voters:
I20260928 21:13:47.432961 12141 raft_consensus.cc:2781] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:13:47.433264 12141 raft_consensus.cc:686] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 1 LEADER]: Becoming Leader. State: Replica: d99d3c4456414a028a3f018ca7a04382, State: Running, Role: LEADER
I20260928 21:13:47.433641 12141 consensus_queue.cc:237] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } }
I20260928 21:13:47.437638 11630 catalog_manager.cc:5495] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 reported cstate change: term changed from 0 to 1, leader changed from <none> to d99d3c4456414a028a3f018ca7a04382 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d99d3c4456414a028a3f018ca7a04382" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d99d3c4456414a028a3f018ca7a04382" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40663 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 } health_report { overall_health: UNKNOWN } } }
I20260928 21:13:47.439388 12129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.416593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51091 (local address 127.0.98.1:41593)
0928 21:13:47.416727 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.416732 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.418537 (+ 1805us) server_negotiation.cc:407] Connection header received
0928 21:13:47.418712 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.418715 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.418757 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.418801 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:13:47.424902 (+ 6101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.425650 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.438075 (+ 12425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.438630 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.438720 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.438783 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.439171 (+ 388us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.439262 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260928 21:13:47.439970 12144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.416249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.98.2:51091)
0928 21:13:47.418447 (+ 2198us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.418464 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:13:47.418604 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:13:47.418932 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:13:47.418935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:13:47.418949 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:13:47.419051 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.419057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.436718 (+ 17661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.436721 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.437898 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:13:47.437910 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.438896 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.438902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:13:47.439015 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.439045 (+ 30us) client_negotiation.cc:769] Sending connection context
0928 21:13:47.439730 (+ 685us) client_negotiation.cc:240] Negotiation successful
0928 21:13:47.439812 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":98,"threads_started":1}
I20260928 21:13:47.440788 11799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8b1db85da0401eac86eae605b71527" candidate_uuid: "d99d3c4456414a028a3f018ca7a04382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ababe079d1d04afd829bd891cc513e2d" is_pre_election: true
I20260928 21:13:47.440999 11798 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a8b1db85da0401eac86eae605b71527" candidate_uuid: "d99d3c4456414a028a3f018ca7a04382" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ababe079d1d04afd829bd891cc513e2d"
I20260928 21:13:47.441129 11799 raft_consensus.cc:2454] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d99d3c4456414a028a3f018ca7a04382 in term 0.
I20260928 21:13:47.468475 12130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.464247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44542 (local address 127.0.98.2:40663)
0928 21:13:47.464406 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0928 21:13:47.464411 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:13:47.464455 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:13:47.464622 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:13:47.464629 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:13:47.464705 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:13:47.464816 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:13:47.465119 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.465697 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.466637 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:13:47.467102 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:13:47.467251 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:13:47.467609 (+ 358us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:13:47.468055 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:13:47.468203 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:13:47.468245 (+ 42us) server_negotiation.cc:299] Negotiation successful
0928 21:13:47.468303 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:13:47.476161 12050 raft_consensus.cc:1261] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 1 FOLLOWER]: Refusing update from remote peer d99d3c4456414a028a3f018ca7a04382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:13:47.476994 12141 consensus_queue.cc:1035] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "640ca2bbb6074fb3ac108251f27e0742" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:13:47.477460 11798 raft_consensus.cc:3037] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:13:47.478926 11798 raft_consensus.cc:1261] T 1a8b1db85da0401eac86eae605b71527 P ababe079d1d04afd829bd891cc513e2d [term 1 FOLLOWER]: Refusing update from remote peer d99d3c4456414a028a3f018ca7a04382: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:13:47.479624 12141 consensus_queue.cc:1035] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ababe079d1d04afd829bd891cc513e2d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:13:47.493315 12148 mvcc.cc:205] Tried to move back new op lower bound from 7334420592534802432 to 7334420592372125696. Current Snapshot: MvccSnapshot[applied={T|T < 7334420592534802432}]
I20260928 21:13:47.495011 12151 mvcc.cc:205] Tried to move back new op lower bound from 7334420592534802432 to 7334420592372125696. Current Snapshot: MvccSnapshot[applied={T|T < 7334420592534802432}]
I20260928 21:13:47.497123 12150 mvcc.cc:205] Tried to move back new op lower bound from 7334420592534802432 to 7334420592372125696. Current Snapshot: MvccSnapshot[applied={T|T < 7334420592534802432}]
I20260928 21:13:47.515929 11631 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:13:47.516129 11631 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39958:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:13:47.519229 11631 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cf8ecbf8289e4c6eb36a5b95ac78c4be: Sending DeleteTablet for 3 replicas of tablet 1a8b1db85da0401eac86eae605b71527
I20260928 21:13:47.520237 11779 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8b1db85da0401eac86eae605b71527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:13:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38270
I20260928 21:13:47.521152 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8b1db85da0401eac86eae605b71527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:13:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44536
I20260928 21:13:47.522025 12038 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a8b1db85da0401eac86eae605b71527 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:13:47 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42650
I20260928 21:13:47.523217 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 11712
I20260928 21:13:47.528633 12161 tablet_replica.cc:330] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: stopping tablet replica
I20260928 21:13:47.529281 12160 tablet_replica.cc:330] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: stopping tablet replica
I20260928 21:13:47.529035 12161 raft_consensus.cc:2229] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:13:47.529736 12160 raft_consensus.cc:2229] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:13:47.531196 12160 raft_consensus.cc:2258] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:13:47.531358 12161 raft_consensus.cc:2258] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:13:47.533125 12161 ts_tablet_manager.cc:1905] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:13:47.534010 12160 ts_tablet_manager.cc:1905] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:13:47.537595 12160 ts_tablet_manager.cc:1918] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:13:47.540062 12161 ts_tablet_manager.cc:1918] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:13:47.540164 12161 log.cc:1196] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/1a8b1db85da0401eac86eae605b71527
I20260928 21:13:47.540480 12161 ts_tablet_manager.cc:1939] T 1a8b1db85da0401eac86eae605b71527 P 640ca2bbb6074fb3ac108251f27e0742: Deleting consensus metadata
I20260928 21:13:47.541850 11616 catalog_manager.cc:4841] TS 640ca2bbb6074fb3ac108251f27e0742 (127.0.98.3:43407): tablet 1a8b1db85da0401eac86eae605b71527 (table test-table [id=6ec85d4b1f5e48519bcc650df9b3a86e]) successfully deleted
I20260928 21:13:47.542284 12160 log.cc:1196] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/1a8b1db85da0401eac86eae605b71527
W20260928 21:13:47.542264 11618 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41593 (error 108)
I20260928 21:13:47.542714 12160 ts_tablet_manager.cc:1939] T 1a8b1db85da0401eac86eae605b71527 P d99d3c4456414a028a3f018ca7a04382: Deleting consensus metadata
I20260928 21:13:47.543186 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 11851
I20260928 21:13:47.549156 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.546034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41593 (local address 127.0.0.1:38284)
0928 21:13:47.548976 (+ 2942us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.549061 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2823}
W20260928 21:13:47.549367 11618 catalog_manager.cc:4584] TS ababe079d1d04afd829bd891cc513e2d (127.0.98.1:41593): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a8b1db85da0401eac86eae605b71527: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41593: connect: Connection refused (error 111)
I20260928 21:13:47.555147 12131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:13:47.554860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40663 (local address 127.0.0.1:44544)
0928 21:13:47.555009 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0928 21:13:47.555067 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40663: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 21:13:47.555290 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 11986
I20260928 21:13:47.562508 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 11599
I20260928 21:13:47.568948 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:13:47.903417 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:13:47.903566 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:13:48.249123 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:13:48.249 UTC [11185] LOG: received smart shutdown request
2026-09-28 21:13:48.252 UTC [11185] LOG: background worker "logical replication launcher" (PID 11194) exited with exit code 1
2026-09-28 21:13:48.253 UTC [11189] LOG: shutting down
2026-09-28 21:13:48.427 UTC [11185] LOG: database system is shut down
2026-09-28T21:13:48Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47238 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:13:48 dist-test-slave-v7xz krb5kdc[12171](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:13:48 dist-test-slave-v7xz krb5kdc[12171](info): set up 2 sockets
Sep 28 21:13:48 dist-test-slave-v7xz krb5kdc[12171](info): commencing operation
krb5kdc: starting...
W20260928 21:13:49.580858 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:13:49 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:13:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:13:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260928 21:13:49.743826 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task8iL2mC/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/postgres -l logfile start
2026-09-28 21:13:52.250 UTC [12201] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-28 21:13:52.250 UTC [12201] LOG: listening on IPv4 address "127.0.98.20", port 60443
2026-09-28 21:13:52.254 UTC [12201] LOG: listening on Unix socket "/tmp/.s.PGSQL.60443"
2026-09-28 21:13:52.275 UTC [12204] LOG: database system was shut down at 2026-09-28 21:13:50 UTC
2026-09-28 21:13:52.279 UTC [12201] LOG: database system is ready to accept connections
I20260928 21:13:53.228955 392 mini_postgres.cc:96] Postgres bound to 60443
2026-09-28 21:13:53.235 UTC [12212] FATAL: database "slave" does not exist
127.0.98.20:60443 - accepting connections
I20260928 21:13:53.235762 392 mini_ranger.cc:161] Starting Ranger...
I20260928 21:13:53.250476 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260928 21:13:53.726411 392 mini_ranger.cc:87] Created ranger Postgres database
I20260928 21:13:53.726483 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-admin
2026-09-28 21:13:54,011 [I] DB FLAVOR :POSTGRES
2026-09-28 21:13:54,011 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:13:54,011 [I] Checking connection..
2026-09-28 21:13:54,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-28 21:13:54,267 [I] Checking connection passed.
2026-09-28 21:13:54,267 [I] --------- Verifying version history table ---------
2026-09-28 21:13:54,267 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:13:54,526 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:13:54,527 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:13:54,783 [I] Table x_db_version_h does not exist in database ranger
2026-09-28 21:13:54,783 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-28 21:13:54,783 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-28 21:13:55.032 UTC [12276] WARNING: there is no transaction in progress
2026-09-28 21:13:55,039 [I] create_dbversion_catalog.sql file imported successfully
2026-09-28 21:13:55,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-28 21:13:55,313 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-28 21:13:55,313 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:13:55,314 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-28 21:13:55,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-28 21:13:55,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-v7xz.c.gcp-upstream.internal','N') ;"
2026-09-28 21:13:56,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:13:56,281 [I] Table x_portal_user does not exist in database ranger
2026-09-28 21:13:56,282 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:13:56,538 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-28 21:13:56,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:13:56,791 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:13:57,017 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-28 21:13:57,017 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-28 21:13:57.935 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:57.952 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:57.970 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:57.987 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.006 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.027 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.093 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.100 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.115 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.125 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.136 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.150 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.160 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.167 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.173 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58.181 UTC [12384] WARNING: there is no transaction in progress
2026-09-28 21:13:58,859 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-28 21:13:58,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-28 21:13:59,089 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-28 21:13:59,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-28 21:13:59,332 [I] Table x_portal_user already exists in database 'ranger'
2026-09-28 21:13:59,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-28 21:13:59,564 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-28 21:13:59,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-28 21:13:59,792 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-28 21:14:00,025 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-v7xz.c.gcp-upstream.internal';"
2026-09-28 21:14:00,251 [I] CORE_DB_SCHEMA import status has been updated
2026-09-28 21:14:00,251 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-28 21:14:00,501 [I] DB_PATCHES have already been applied
I20260928 21:14:00.506783 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-admin:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task8iL2mC/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260928 21:14:00.506867 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260928 21:14:00.510243 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37053
Sep 28, 2026 9:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 28, 2026 9:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 28, 2026 9:14:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 28, 2026 9:14:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 28, 2026 9:14:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260928 21:14:01.626194 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.390s user 0.002s sys 0.007s
Sep 28, 2026 9:14:21 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 28, 2026 9:14:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 28, 2026 9:14:21 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 28, 2026 9:14:21 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 28, 2026 9:14:22 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260928 21:14:22.766178 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 28 21:14:22 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630062, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260928 21:14:22.847993 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260928 21:14:22.857383 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260928 21:14:23.712738 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260928 21:14:23.712859 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms
I20260928 21:14:23.712888 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:60443
2026-09-28 21:14:23,891 [I] DB FLAVOR :POSTGRES
2026-09-28 21:14:23,892 [I] --------- Verifying Ranger DB connection ---------
2026-09-28 21:14:23,892 [I] Checking connection
2026-09-28 21:14:23,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-28 21:14:24,121 [I] connection success
2026-09-28 21:14:24,121 [I] --------- Verifying Ranger DB tables ---------
2026-09-28 21:14:24,121 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-28 21:14:24,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-28 21:14:24,350 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-28 21:14:24,351 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-28 21:14:24,351 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-28 21:14:24,351 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:60443/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-28 21:14:24,629 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260928 21:14:25.089025 392 mini_ranger_kms.cc:310] Created kms service
I20260928 21:14:25.262471 392 mini_ranger_kms.cc:326] Created kudu user
I20260928 21:14:25.306675 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 28 21:14:25 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630062, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:25.703336 392 mini_ranger_kms.cc:384] Added ranger policy
I20260928 21:14:25.703613 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms:/tmp/dist-test-task8iL2mC/build/debug/bin/postgresql.jar:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task8iL2mC/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260928 21:14:25.703637 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260928 21:14:25.707383 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12564
I20260928 21:14:25.707475 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52745
Sep 28, 2026 9:14:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 28, 2026 9:14:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 28, 2026 9:14:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 28, 2026 9:14:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 28, 2026 9:14:27 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 28, 2026 9:14:27 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260928 21:14:27.965255 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.117s user 0.000s sys 0.163s
I20260928 21:14:27.965380 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260928 21:14:27.965461 392 mini_ranger_kms.cc:401] 127.0.98.20:52745/kms/v1/keys
Sep 28 21:14:27 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630067, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 28 21:14:28 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630067, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 21:14:28 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630067, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 21:14:31 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:14:31.101323 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34163
--ranger_config_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:31.213322 12615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:31.213624 12615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:31.213696 12615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:31.217731 12615 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:31.217813 12615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:31.217850 12615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:31.217909 12615 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:31.217948 12615 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:31.223459 12615 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52745/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34163
--ranger_config_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12615
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:31.224496 12615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:31.225541 12615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:31.234020 12621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:31.234054 12620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:31.234051 12615 server_base.cc:1034] running on GCE node
W20260928 21:14:31.234076 12623 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:31.234779 12615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:31.235900 12615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:31.237087 12615 hybrid_clock.cc:648] HybridClock initialized: now 1790630071237074 us; error 28 us; skew 500 ppm
Sep 28 21:14:31 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:31.245312 12615 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:31.246210 12615 webserver.cc:460] Webserver started at http://127.0.98.62:44845/ using document root <none> and password file <none>
I20260928 21:14:31.246771 12615 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:31.246853 12615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:31.247072 12615 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:14:31 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:31.295538 12615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "fe4c66fee4c34ca18cab2a09d8b7918b"
format_stamp: "Formatted at 2026-09-28 21:14:31 on dist-test-slave-v7xz"
server_key: "75487cabebc15b9291e3865858a6f99f"
server_key_iv: "37107c4ea12eec8d6e6eb3ecadfa163e"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:31.296270 12615 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "fe4c66fee4c34ca18cab2a09d8b7918b"
format_stamp: "Formatted at 2026-09-28 21:14:31 on dist-test-slave-v7xz"
server_key: "75487cabebc15b9291e3865858a6f99f"
server_key_iv: "37107c4ea12eec8d6e6eb3ecadfa163e"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:31.300545 12615 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:14:31.318948 12630 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:31.320482 12615 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:31.320652 12615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "fe4c66fee4c34ca18cab2a09d8b7918b"
format_stamp: "Formatted at 2026-09-28 21:14:31 on dist-test-slave-v7xz"
server_key: "75487cabebc15b9291e3865858a6f99f"
server_key_iv: "37107c4ea12eec8d6e6eb3ecadfa163e"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:31.320803 12615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:31.340380 12615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:31.343343 12615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:31.343546 12615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:31.351891 12615 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34163
I20260928 21:14:31.351934 12682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34163 every 8 connection(s)
I20260928 21:14:31.353034 12615 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:31.356202 12683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:31.359696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 12615
I20260928 21:14:31.359802 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:14:31.360086 392 external_mini_cluster.cc:1428] Setting key 5f625681c1eb71b8bbc9ac72728cd3b5
I20260928 21:14:31.362295 12683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Bootstrap starting.
I20260928 21:14:31.364993 12683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:31.365824 12683 log.cc:824] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:31.368060 12683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: No bootstrap required, opened a new log
Sep 28 21:14:31 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:31.371979 12683 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:31.372242 12683 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:31.372334 12683 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4c66fee4c34ca18cab2a09d8b7918b, State: Initialized, Role: FOLLOWER
I20260928 21:14:31.373009 12683 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:31.373186 12683 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:14:31.373258 12683 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:14:31.373410 12683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:31.374377 12683 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:31.374747 12683 leader_election.cc:302] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4c66fee4c34ca18cab2a09d8b7918b; no voters:
I20260928 21:14:31.375160 12683 leader_election.cc:288] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:14:31.375236 12688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:31.375427 12688 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 LEADER]: Becoming Leader. State: Replica: fe4c66fee4c34ca18cab2a09d8b7918b, State: Running, Role: LEADER
I20260928 21:14:31.375783 12688 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:31.375980 12686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:31.361606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53884 (local address 127.0.98.62:34163)
0928 21:14:31.362037 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:31.362059 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:31.362102 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:14:31.362797 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:31.362817 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:31.363175 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:31.363472 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:31.364409 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:31.365561 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:31.366774 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:31.367252 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:31.367536 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:31.370306 (+ 2770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:31.370314 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:31.370335 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:31.370374 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:31.372520 (+ 2146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:31.373174 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:31.373180 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:31.373187 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:31.373353 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:31.373711 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:31.373715 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:31.373717 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:31.374142 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:31.374349 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:31.374655 (+ 306us) server_negotiation.cc:299] Negotiation successful
0928 21:14:31.374965 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":151,"threads_started":1}
I20260928 21:14:31.376498 12683 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:31.378803 12683 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:14:31.378815 12690 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4c66fee4c34ca18cab2a09d8b7918b. Latest consensus state: current_term: 1 leader_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260928 21:14:31.378965 12690 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:31.379351 12702 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:31.379940 12689 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260928 21:14:31.380043 12689 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:31.384644 12702 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:31.390693 12702 catalog_manager.cc:1347] Generated new cluster ID: 0dfc88508b1d433eb25e8b05f89bec90
I20260928 21:14:31.390769 12702 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:14:31.403028 12702 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:14:31.404434 12702 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:14:31.419066 12702 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Generated new TSK 0
I20260928 21:14:31.419999 12702 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:14:32.124720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:32.269480 12728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:32.269863 12728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:32.269950 12728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:32.274475 12728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:32.274618 12728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:32.274730 12728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:14:32.280754 12728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:32.281941 12728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:32.283026 12728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:14:32.292459 12728 server_base.cc:1034] running on GCE node
W20260928 21:14:32.292447 12736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:32.292762 12738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:32.292946 12735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:32.293434 12728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:32.294296 12728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:32.295498 12728 hybrid_clock.cc:648] HybridClock initialized: now 1790630072295467 us; error 46 us; skew 500 ppm
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:32.299201 12728 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:14:32.300277 12728 webserver.cc:460] Webserver started at http://127.0.98.1:34711/ using document root <none> and password file <none>
I20260928 21:14:32.300999 12728 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:32.301126 12728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:32.301364 12728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:32.317538 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "29d27625e448480a8ae1c0c5ea09cb6c"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "f87806362da37b5705a5365070ca77df"
server_key_iv: "628db449c1bbb529c150355ec50afae5"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:32.318189 12728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "29d27625e448480a8ae1c0c5ea09cb6c"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "f87806362da37b5705a5365070ca77df"
server_key_iv: "628db449c1bbb529c150355ec50afae5"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:32.322324 12728 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:14:32.361315 12745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:32.363503 12728 fs_manager.cc:725] Time spent opening block manager: real 0.026s user 0.002s sys 0.000s
I20260928 21:14:32.363680 12728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "29d27625e448480a8ae1c0c5ea09cb6c"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "f87806362da37b5705a5365070ca77df"
server_key_iv: "628db449c1bbb529c150355ec50afae5"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:32.363835 12728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:32.392434 12728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:32.396119 12728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:32.396332 12728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:32.397023 12728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:32.397943 12728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:32.398015 12728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:32.398104 12728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:32.398160 12728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:32.409587 12728 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37757
I20260928 21:14:32.410518 12728 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:14:32.411679 12860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37757 every 8 connection(s)
I20260928 21:14:32.414188 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 12728
I20260928 21:14:32.414311 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:14:32.414614 392 external_mini_cluster.cc:1428] Setting key d2522c1c0789517d2f8f1c7a5ae05df5
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:32.427241 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:32.412515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40377 (local address 127.0.98.62:34163)
0928 21:14:32.412817 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:32.412863 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:32.413711 (+ 848us) server_negotiation.cc:407] Connection header received
0928 21:14:32.414987 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:32.414993 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:32.415052 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:32.415125 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:32.416757 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.417682 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.419525 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.419776 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.419894 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:32.422391 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:32.422395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:32.422398 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:32.422424 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:32.424380 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:32.425160 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:32.425165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:32.425167 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:32.425261 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:32.425700 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:32.425704 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:32.425706 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:32.425959 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:32.426054 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:32.426822 (+ 768us) server_negotiation.cc:299] Negotiation successful
0928 21:14:32.426965 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:14:32.428969 12864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:32.413068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.1:40377)
0928 21:14:32.413544 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:32.413608 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:32.414727 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:32.415537 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:32.415546 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:32.415903 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:32.416572 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:32.416606 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.417856 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.417861 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:32.419384 (+ 1523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:32.419395 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.419942 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.419946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:32.420135 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:32.420781 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:32.420799 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:32.422212 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:32.424578 (+ 2366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:32.424586 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:32.424608 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:32.425021 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:32.425389 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:32.425395 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:32.425398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:32.425577 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:32.426140 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:32.426148 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:32.426509 (+ 361us) client_negotiation.cc:769] Sending connection context
0928 21:14:32.427609 (+ 1100us) client_negotiation.cc:240] Negotiation successful
0928 21:14:32.427997 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:14:32.430444 12861 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260928 21:14:32.430712 12861 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:32.431293 12861 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:32.433101 12647 ts_manager.cc:194] Registered new tserver with Master: 29d27625e448480a8ae1c0c5ea09cb6c (127.0.98.1:37757)
I20260928 21:14:32.434759 12647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40377
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:14:32.448788 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260928 21:14:32.576571 12867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:32.577095 12867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:32.577214 12867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:32.583482 12867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:32.583564 12867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:32.583652 12867 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:14:32.589632 12867 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:32.591053 12867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:32.592310 12867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:14:32.603286 12867 server_base.cc:1034] running on GCE node
W20260928 21:14:32.602991 12873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:32.602988 12874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:32.603876 12876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:32.604405 12867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:32.605233 12867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:32.606386 12867 hybrid_clock.cc:648] HybridClock initialized: now 1790630072606323 us; error 82 us; skew 500 ppm
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:32.609413 12867 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:14:32.610402 12867 webserver.cc:460] Webserver started at http://127.0.98.2:33211/ using document root <none> and password file <none>
I20260928 21:14:32.610989 12867 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:32.611050 12867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:32.611260 12867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:32.632110 12867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "8a35bc90a18b4f378caad4b7e153c9cf"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "77121fae8e013ffdf89c841d681b84e4"
server_key_iv: "d22aca8f9cb4b8c5270375669be41e6c"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:32.633242 12867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8a35bc90a18b4f378caad4b7e153c9cf"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "77121fae8e013ffdf89c841d681b84e4"
server_key_iv: "d22aca8f9cb4b8c5270375669be41e6c"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:32.639079 12867 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260928 21:14:32.674659 12883 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:32.676473 12867 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.004s sys 0.000s
I20260928 21:14:32.676614 12867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "8a35bc90a18b4f378caad4b7e153c9cf"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "77121fae8e013ffdf89c841d681b84e4"
server_key_iv: "d22aca8f9cb4b8c5270375669be41e6c"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:32.676749 12867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:32.712486 12867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:32.716575 12867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:32.716866 12867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:32.717617 12867 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:32.718719 12867 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:32.718832 12867 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:32.718933 12867 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:32.719000 12867 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:32.733314 12867 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36515
I20260928 21:14:32.734522 12867 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:14:32.738529 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 12867
I20260928 21:14:32.738654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:14:32.738989 392 external_mini_cluster.cc:1428] Setting key 5d383584a42b15d7d2b6ae374231aece
I20260928 21:14:32.735402 12996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36515 every 8 connection(s)
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:32.759699 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:32.739257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50533 (local address 127.0.98.62:34163)
0928 21:14:32.739371 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:32.739375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:32.739391 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:14:32.744379 (+ 4988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:32.744383 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:32.744456 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:32.744510 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:32.746425 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.747266 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.749012 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.749372 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.749462 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:32.752618 (+ 3156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:32.752621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:32.752624 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:32.752655 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:32.754460 (+ 1805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:32.757591 (+ 3131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:32.757595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:32.757598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:32.757705 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:32.758167 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:32.758171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:32.758173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:32.758451 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:32.758544 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:32.759336 (+ 792us) server_negotiation.cc:299] Negotiation successful
0928 21:14:32.759509 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:14:32.761576 12999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:32.738117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.2:50533)
0928 21:14:32.738747 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:32.738815 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:32.744088 (+ 5273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:32.744986 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:32.744997 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:32.745364 (+ 367us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:32.746177 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:32.746199 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.747457 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.747460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:32.748874 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:32.748884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:32.749676 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:32.749680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:32.749878 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:32.750601 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:32.750624 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:32.752358 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:32.756967 (+ 4609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:32.756982 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:32.757005 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:32.757448 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:32.757824 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:32.757828 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:32.757832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:32.758047 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:32.758620 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:32.758630 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:32.759010 (+ 380us) client_negotiation.cc:769] Sending connection context
0928 21:14:32.760223 (+ 1213us) client_negotiation.cc:240] Negotiation successful
0928 21:14:32.760589 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":156,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:14:32.762990 12997 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260928 21:14:32.763249 12997 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:32.763857 12997 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:32.765174 12647 ts_manager.cc:194] Registered new tserver with Master: 8a35bc90a18b4f378caad4b7e153c9cf (127.0.98.2:36515)
I20260928 21:14:32.766016 12647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50533
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:14:32.785269 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:32.943075 13002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:32.943521 13002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:32.943625 13002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:32.951001 13002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:32.951098 13002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:32.951213 13002 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:14:32.960285 13002 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:32.961853 13002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:32.963304 13002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260928 21:14:32.977396 13008 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:32.978214 13007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:32.982167 13010 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:32.982142 13002 server_base.cc:1034] running on GCE node
I20260928 21:14:32.982849 13002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:32.983826 13002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:32.985015 13002 hybrid_clock.cc:648] HybridClock initialized: now 1790630072984946 us; error 80 us; skew 500 ppm
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:32.992555 13002 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:14:32.993935 13002 webserver.cc:460] Webserver started at http://127.0.98.3:38041/ using document root <none> and password file <none>
I20260928 21:14:32.994699 13002 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:32.994793 13002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:32.995052 13002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 28 21:14:32 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:33.029410 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "7ddd9117db324a55affc164bb9332c02"
format_stamp: "Formatted at 2026-09-28 21:14:33 on dist-test-slave-v7xz"
server_key: "a36155ac884f7353c784f8cd102eb35c"
server_key_iv: "d89b9822a36d3362b1621eb958a04691"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:33.041433 13002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7ddd9117db324a55affc164bb9332c02"
format_stamp: "Formatted at 2026-09-28 21:14:33 on dist-test-slave-v7xz"
server_key: "a36155ac884f7353c784f8cd102eb35c"
server_key_iv: "d89b9822a36d3362b1621eb958a04691"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:33.054661 13002 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.000s
I20260928 21:14:33.081305 13017 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.083303 13002 fs_manager.cc:725] Time spent opening block manager: real 0.009s user 0.003s sys 0.000s
I20260928 21:14:33.083463 13002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "7ddd9117db324a55affc164bb9332c02"
format_stamp: "Formatted at 2026-09-28 21:14:33 on dist-test-slave-v7xz"
server_key: "a36155ac884f7353c784f8cd102eb35c"
server_key_iv: "d89b9822a36d3362b1621eb958a04691"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:33.083604 13002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:33.111938 13002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:33.115700 13002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:33.115906 13002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:33.116670 13002 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:33.117802 13002 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:33.117871 13002 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.117935 13002 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:33.117970 13002 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.133555 13002 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44419
I20260928 21:14:33.134825 13002 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:14:33.135481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13002
I20260928 21:14:33.135629 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:14:33.135404 13130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44419 every 8 connection(s)
I20260928 21:14:33.135969 392 external_mini_cluster.cc:1428] Setting key 894b7f86a2655979edaed2e73a049976
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:33.157208 12863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:33.141015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40661 (local address 127.0.98.62:34163)
0928 21:14:33.141136 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:33.141141 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:33.141698 (+ 557us) server_negotiation.cc:407] Connection header received
0928 21:14:33.142973 (+ 1275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:33.142977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:33.143030 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:33.143080 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:33.144813 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.145691 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.147637 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.147980 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.148054 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:33.152650 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:33.152654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:33.152658 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:33.152688 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:33.154519 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:33.155198 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:33.155203 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:33.155205 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:33.155295 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:33.155770 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:33.155774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:33.155776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:33.156014 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:33.156087 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:33.156888 (+ 801us) server_negotiation.cc:299] Negotiation successful
0928 21:14:33.157029 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:14:33.158988 13133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:33.139842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.3:40661)
0928 21:14:33.141544 (+ 1702us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:33.141605 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:33.142717 (+ 1112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:33.143522 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:33.143532 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:33.143901 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:33.144636 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:33.144657 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.145851 (+ 1194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.145855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:33.147495 (+ 1640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:33.147507 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.148263 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.148267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:33.148481 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:33.150561 (+ 2080us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:33.150663 (+ 102us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:33.152419 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:33.154680 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:33.154687 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:33.154707 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:33.155074 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:33.155454 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:33.155458 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:33.155460 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:33.155653 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:33.156174 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:33.156181 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:33.156557 (+ 376us) client_negotiation.cc:769] Sending connection context
0928 21:14:33.157456 (+ 899us) client_negotiation.cc:240] Negotiation successful
0928 21:14:33.157841 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1461,"thread_start_us":166,"threads_started":1}
I20260928 21:14:33.160487 13131 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260928 21:14:33.160748 13131 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:33.161427 13131 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:33.163530 12647 ts_manager.cc:194] Registered new tserver with Master: 7ddd9117db324a55affc164bb9332c02 (127.0.98.3:44419)
I20260928 21:14:33.164278 12647 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40661
I20260928 21:14:33.176491 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:14:33.177641 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 12728
I20260928 21:14:33.186910 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 12867
I20260928 21:14:33.194353 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13002
I20260928 21:14:33.202876 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 12615
I20260928 21:14:33.212265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--webserver_interface=127.0.98.62
--webserver_port=44845
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34163
--ranger_config_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:33.323688 13134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:33.323975 13134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:33.324016 13134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:33.328302 13134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:33.328373 13134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:33.328392 13134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:33.328409 13134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:33.328433 13134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:33.334268 13134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34163
--ranger_config_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=44845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13134
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:33.335489 13134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:33.336551 13134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:33.345587 13139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:33.346095 13134 server_base.cc:1034] running on GCE node
W20260928 21:14:33.346091 13140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:33.346357 13142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:33.347113 13134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:33.348739 13134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:33.349954 13134 hybrid_clock.cc:648] HybridClock initialized: now 1790630073349927 us; error 53 us; skew 500 ppm
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:33.353005 13134 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:33.354061 13134 webserver.cc:460] Webserver started at http://127.0.98.62:44845/ using document root <none> and password file <none>
I20260928 21:14:33.354622 13134 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:33.354689 13134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:33.418850 13134 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:33.421537 13150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.422915 13134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:14:33.423086 13134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "fe4c66fee4c34ca18cab2a09d8b7918b"
format_stamp: "Formatted at 2026-09-28 21:14:31 on dist-test-slave-v7xz"
server_key: "75487cabebc15b9291e3865858a6f99f"
server_key_iv: "37107c4ea12eec8d6e6eb3ecadfa163e"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:33.423671 13134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:33.442215 13134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:33.445307 13134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:33.445497 13134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:33.452975 13202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34163 every 8 connection(s)
I20260928 21:14:33.452976 13134 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34163
I20260928 21:14:33.454178 13134 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:33.458524 13203 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:14:33.459749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13134
I20260928 21:14:33.460250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37757
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34163
--webserver_port=34711
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:14:33.460605 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Bootstrap starting.
I20260928 21:14:33.477844 13203 log.cc:824] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:33.484601 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:14:33.485059 13203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Bootstrap complete.
I20260928 21:14:33.489164 13203 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:33.489559 13203 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe4c66fee4c34ca18cab2a09d8b7918b, State: Initialized, Role: FOLLOWER
I20260928 21:14:33.490047 13203 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:33.490171 13203 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:14:33.490260 13203 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:14:33.490355 13203 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:14:33.492224 13203 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:33.492560 13203 leader_election.cc:302] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe4c66fee4c34ca18cab2a09d8b7918b; no voters:
I20260928 21:14:33.492951 13203 leader_election.cc:288] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:14:33.493036 13207 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:14:33.493227 13207 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [term 2 LEADER]: Becoming Leader. State: Replica: fe4c66fee4c34ca18cab2a09d8b7918b, State: Running, Role: LEADER
I20260928 21:14:33.493520 13207 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260928 21:14:33.494170 13203 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:33.494854 13208 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260928 21:14:33.494975 13208 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:33.495657 13214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:33.495776 13209 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe4c66fee4c34ca18cab2a09d8b7918b. Latest consensus state: current_term: 2 leader_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe4c66fee4c34ca18cab2a09d8b7918b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260928 21:14:33.495898 13209 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:33.498997 13214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:33.499527 13203 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260928 21:14:33.499758 13214 catalog_manager.cc:1259] Loaded cluster ID: 0dfc88508b1d433eb25e8b05f89bec90
I20260928 21:14:33.499856 13214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:14:33.505007 13214 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:14:33.506803 13214 catalog_manager.cc:5879] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Loaded TSK: 0
I20260928 21:14:33.508373 13214 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:14:33.583493 13205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:33.583920 13205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:33.584029 13205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:33.588752 13205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:33.588886 13205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:33.588992 13205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:14:33.595656 13205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=34711
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:33.597229 13205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:33.598495 13205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:33.611452 13243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:33.612430 13241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:33.612509 13240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:33.613055 13205 server_base.cc:1034] running on GCE node
I20260928 21:14:33.613631 13205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:33.614405 13205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:33.617753 13205 hybrid_clock.cc:648] HybridClock initialized: now 1790630073617330 us; error 423 us; skew 500 ppm
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:33.621001 13205 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:14:33.622129 13205 webserver.cc:460] Webserver started at http://127.0.98.1:34711/ using document root <none> and password file <none>
I20260928 21:14:33.622749 13205 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:33.622864 13205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:33.656577 13205 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:14:33.661759 13251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.663498 13205 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260928 21:14:33.663668 13205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "29d27625e448480a8ae1c0c5ea09cb6c"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "f87806362da37b5705a5365070ca77df"
server_key_iv: "628db449c1bbb529c150355ec50afae5"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:33.664445 13205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:33.686034 13205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:33.689332 13205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:33.689554 13205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:33.690233 13205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:33.691157 13205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:33.691231 13205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.691293 13205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:33.691321 13205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.701943 13205 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37757
I20260928 21:14:33.702090 13364 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37757 every 8 connection(s)
I20260928 21:14:33.703315 13205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:14:33.712378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13205
I20260928 21:14:33.712934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:36515
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34163
--webserver_port=33211
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:33.740918 13367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:33.711575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.1:33791)
0928 21:14:33.713188 (+ 1613us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:33.713251 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:33.714289 (+ 1038us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:33.722328 (+ 8039us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:33.722342 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:33.722778 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:33.723634 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:33.723661 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.726276 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.726280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:33.727726 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:33.727743 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.728850 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.728854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:33.729056 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:33.729891 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:33.729912 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:33.731541 (+ 1629us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:33.737009 (+ 5468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:33.737019 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:33.737037 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:33.737447 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:33.737907 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:33.737912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:33.737915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:33.738105 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:33.738907 (+ 802us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:33.738917 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:33.739270 (+ 353us) client_negotiation.cc:769] Sending connection context
0928 21:14:33.739479 (+ 209us) client_negotiation.cc:240] Negotiation successful
0928 21:14:33.739828 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1382,"thread_start_us":179,"threads_started":1}
I20260928 21:14:33.743340 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:33.713226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33791 (local address 127.0.98.62:34163)
0928 21:14:33.720543 (+ 7317us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:33.720575 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:33.720642 (+ 67us) server_negotiation.cc:407] Connection header received
0928 21:14:33.721076 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:33.721100 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:33.721670 (+ 570us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:33.723953 (+ 2283us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:33.724301 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.726097 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.727895 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.728634 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.729532 (+ 898us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:33.731836 (+ 2304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:33.731844 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:33.731864 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:33.731899 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:33.736803 (+ 4904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:33.737614 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:33.737621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:33.737627 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:33.737785 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:33.738227 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:33.738230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:33.738235 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:33.738759 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:33.741977 (+ 3218us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:33.742134 (+ 157us) server_negotiation.cc:299] Negotiation successful
0928 21:14:33.742447 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7071,"thread_start_us":130,"threads_started":1}
I20260928 21:14:33.746112 13365 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260928 21:14:33.746372 13365 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:33.746883 13365 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
W20260928 21:14:33.749226 13365 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:14:33.749369 13365 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:33.749567 13365 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:33.750556 13365 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:33.750756 13365 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
W20260928 21:14:33.751569 13365 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:14:33.870055 13369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:33.870379 13369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:33.870481 13369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:33.875258 13369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:33.875365 13369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:33.875463 13369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:14:33.883255 13369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:36515
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33211
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:33.884779 13369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:33.886061 13369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:33.896896 13375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:33.896951 13377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:33.897089 13374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:33.897488 13369 server_base.cc:1034] running on GCE node
I20260928 21:14:33.897897 13369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:33.898617 13369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:33.899811 13369 hybrid_clock.cc:648] HybridClock initialized: now 1790630073899774 us; error 59 us; skew 500 ppm
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:33.903220 13369 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:14:33.904320 13369 webserver.cc:460] Webserver started at http://127.0.98.2:33211/ using document root <none> and password file <none>
I20260928 21:14:33.905098 13369 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:33.905208 13369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:33.925316 13369 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:14:33.928222 13385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.929981 13369 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:14:33.930150 13369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "8a35bc90a18b4f378caad4b7e153c9cf"
format_stamp: "Formatted at 2026-09-28 21:14:32 on dist-test-slave-v7xz"
server_key: "77121fae8e013ffdf89c841d681b84e4"
server_key_iv: "d22aca8f9cb4b8c5270375669be41e6c"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:33.930915 13369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:33.953358 13369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:33.956833 13369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:33.957106 13369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:33.957860 13369 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:33.958977 13369 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:33.959061 13369 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.959134 13369 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:33.959169 13369 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:33.972483 13369 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:36515
I20260928 21:14:33.974073 13369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:14:33.977082 13498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:36515 every 8 connection(s)
I20260928 21:14:33.981925 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13369
I20260928 21:14:33.982481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52745/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44419
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34163
--webserver_port=38041
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:14:33 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630073, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:33.994971 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:33.979290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37955 (local address 127.0.98.62:34163)
0928 21:14:33.979417 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:33.979422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:33.981309 (+ 1887us) server_negotiation.cc:407] Connection header received
0928 21:14:33.982370 (+ 1061us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:33.982373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:33.982440 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:33.982530 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:33.984103 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.984986 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.986771 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.987155 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.987244 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:33.990449 (+ 3205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:33.990453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:33.990456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:33.990489 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:33.992339 (+ 1850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:33.993028 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:33.993033 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:33.993035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:33.993136 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:33.993572 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:33.993575 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:33.993577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:33.993809 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:33.993895 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:33.994650 (+ 755us) server_negotiation.cc:299] Negotiation successful
0928 21:14:33.994791 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:14:33.997272 13501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:33.979570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.2:37955)
0928 21:14:33.981137 (+ 1567us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:33.981182 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:33.982142 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:33.982741 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:33.982752 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:33.983143 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:33.983846 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:33.983869 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.985177 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.985180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:33.986616 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:33.986628 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:33.987487 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:33.987491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:33.987709 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:33.988409 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:33.988432 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:33.990234 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:33.992481 (+ 2247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:33.992487 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:33.992504 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:33.992903 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:33.993251 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:33.993254 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:33.993256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:33.993455 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:33.993963 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:33.993971 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:33.994340 (+ 369us) client_negotiation.cc:769] Sending connection context
0928 21:14:33.995643 (+ 1303us) client_negotiation.cc:240] Negotiation successful
0928 21:14:33.996032 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1353,"thread_start_us":125,"threads_started":1}
I20260928 21:14:33.999161 13499 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260928 21:14:33.999428 13499 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.000061 13499 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
W20260928 21:14:34.001905 13499 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:14:34.002036 13499 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.002231 13499 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:34.003108 13499 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.003270 13499 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
W20260928 21:14:34.004004 13499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:14:34.127458 13502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:34.127879 13502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:34.127966 13502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:34.134536 13502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:34.134658 13502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:34.134761 13502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:14:34.143554 13502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:52745/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44419
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38041
--tserver_master_addrs=127.0.98.62:34163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:34.145166 13502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:34.146641 13502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:34.159042 13509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:34.159053 13508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:34.160452 13511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:34.160720 13502 server_base.cc:1034] running on GCE node
I20260928 21:14:34.161324 13502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:34.162236 13502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:34.163419 13502 hybrid_clock.cc:648] HybridClock initialized: now 1790630074163388 us; error 52 us; skew 500 ppm
Sep 28 21:14:34 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:34.167502 13502 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:14:34.168776 13502 webserver.cc:460] Webserver started at http://127.0.98.3:38041/ using document root <none> and password file <none>
I20260928 21:14:34.169582 13502 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:34.169704 13502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 28 21:14:34 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:34.200569 13502 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.001s
I20260928 21:14:34.204164 13519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:34.205922 13502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260928 21:14:34.206082 13502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "7ddd9117db324a55affc164bb9332c02"
format_stamp: "Formatted at 2026-09-28 21:14:33 on dist-test-slave-v7xz"
server_key: "a36155ac884f7353c784f8cd102eb35c"
server_key_iv: "d89b9822a36d3362b1621eb958a04691"
server_key_version: "kuduclusterkey@0"
I20260928 21:14:34.206713 13502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:34.228790 13502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:34.232506 13502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:34.232738 13502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:34.233440 13502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:34.234531 13502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:34.234619 13502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:34.234687 13502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:34.234720 13502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:34.248718 13502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44419
I20260928 21:14:34.249063 13632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44419 every 8 connection(s)
I20260928 21:14:34.250252 13502 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:14:34.251482 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13502
I20260928 21:14:34.263985 13368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:34.252638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.0.98.62:34163)
0928 21:14:34.252777 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:34.252785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:34.253134 (+ 349us) server_negotiation.cc:407] Connection header received
0928 21:14:34.253365 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:34.253369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:34.253425 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:34.253484 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:34.254629 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:34.255444 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:34.256823 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:34.257231 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:34.257310 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:34.259407 (+ 2097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:34.259412 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:34.259415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:34.259444 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:34.261576 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:34.262140 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:34.262143 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:34.262145 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:34.262232 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:34.262527 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:34.262531 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:34.262533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:34.262748 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:34.262933 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:34.263618 (+ 685us) server_negotiation.cc:299] Negotiation successful
0928 21:14:34.263765 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":54}
Sep 28 21:14:34 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:34.270316 13636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:34.253036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55683 (local address 127.0.98.62:34163)
0928 21:14:34.255210 (+ 2174us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:34.255215 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:34.256870 (+ 1655us) server_negotiation.cc:407] Connection header received
0928 21:14:34.257096 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:34.257099 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:34.257156 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:34.257241 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:34.259068 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:34.259988 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:34.262139 (+ 2151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:34.262527 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:34.262600 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:34.265958 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:34.265963 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:34.265966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:34.265994 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:34.267761 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:34.268320 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:34.268324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:34.268325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:34.268418 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:34.268965 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:34.268968 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:34.268970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:34.269169 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:34.269280 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:34.269949 (+ 669us) server_negotiation.cc:299] Negotiation successful
0928 21:14:34.270083 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1686,"server-negotiator.queue_time_us":2099,"thread_start_us":98,"threads_started":1}
I20260928 21:14:34.271313 13637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:34.254025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.3:55683)
0928 21:14:34.255493 (+ 1468us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:34.255561 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:34.256687 (+ 1126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:34.257594 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:34.257604 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:34.258006 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:34.258840 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:34.258866 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:34.260163 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:34.260168 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:34.262013 (+ 1845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:34.262024 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:34.262817 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:34.262821 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:34.263033 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:34.263665 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:34.263685 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:34.265770 (+ 2085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:34.267894 (+ 2124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:34.267908 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:34.267927 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:34.268209 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:34.268519 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:34.268522 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:34.268524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:34.268859 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:34.269291 (+ 432us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:34.269299 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:34.269605 (+ 306us) client_negotiation.cc:769] Sending connection context
0928 21:14:34.269802 (+ 197us) client_negotiation.cc:240] Negotiation successful
0928 21:14:34.270225 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1226,"thread_start_us":145,"threads_started":1}
I20260928 21:14:34.272699 13633 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260928 21:14:34.273002 13633 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.273551 13633 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
W20260928 21:14:34.275250 13633 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34163 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260928 21:14:34.275390 13633 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.275612 13633 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:34.276471 13633 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.276674 13633 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
W20260928 21:14:34.278600 13633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 28 21:14:34 dist-test-slave-v7xz krb5kdc[12171](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790630074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:34.751816 13365 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:34.752111 13365 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:34.753276 13163 ts_manager.cc:194] Registered new tserver with Master: 29d27625e448480a8ae1c0c5ea09cb6c (127.0.98.1:37757)
I20260928 21:14:34.754529 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33791
I20260928 21:14:35.004256 13499 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:35.004577 13499 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:35.005590 13163 ts_manager.cc:194] Registered new tserver with Master: 8a35bc90a18b4f378caad4b7e153c9cf (127.0.98.2:36515)
I20260928 21:14:35.006407 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37955
Sep 28 21:14:35 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 28 21:14:35 dist-test-slave-v7xz krb5kdc[12171](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260928 21:14:35.278825 13633 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:35.279114 13633 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260928 21:14:35.283839 13163 ts_manager.cc:194] Registered new tserver with Master: 7ddd9117db324a55affc164bb9332c02 (127.0.98.3:44419)
I20260928 21:14:35.284775 13163 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55683
I20260928 21:14:35.288463 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:14:35.302676 13655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.291562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33384 (local address 127.0.98.62:34163)
0928 21:14:35.291901 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.291907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.292147 (+ 240us) server_negotiation.cc:407] Connection header received
0928 21:14:35.292292 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.292296 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.292346 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.292432 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:35.293714 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.295727 (+ 2013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.296808 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.297173 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.297263 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.298934 (+ 1671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:35.298938 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:35.298941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:35.298967 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:35.300661 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:35.301153 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:35.301156 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:35.301159 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:35.301240 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:35.301499 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:35.301502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:35.301504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:35.301707 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:35.301822 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.302393 (+ 571us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.302505 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257,"thread_start_us":75,"threads_started":1}
I20260928 21:14:35.307052 13163 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:14:35.311861 13163 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:14:35.341763 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.326252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39272 (local address 127.0.98.3:44419)
0928 21:14:35.328323 (+ 2071us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.328331 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.328428 (+ 97us) server_negotiation.cc:407] Connection header received
0928 21:14:35.328511 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.328517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.331762 (+ 3245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.332628 (+ 866us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:35.336716 (+ 4088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.337788 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.339842 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.340638 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.340725 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.340929 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.341475 (+ 546us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.341574 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1345,"thread_start_us":81,"threads_started":1}
I20260928 21:14:35.341966 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.330312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44092 (local address 127.0.98.2:36515)
0928 21:14:35.331460 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.331468 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.334530 (+ 3062us) server_negotiation.cc:407] Connection header received
0928 21:14:35.334752 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.334758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.334985 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.335108 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:35.335631 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.337130 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.339585 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.341190 (+ 1605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.341262 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.341402 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.341718 (+ 316us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.341822 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":104,"threads_started":1}
I20260928 21:14:35.342267 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.326122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44419 (local address 127.0.0.1:39272)
0928 21:14:35.327139 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.327191 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:35.327414 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:35.336419 (+ 9005us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:35.336421 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:35.336440 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:35.336524 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.336530 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.337928 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.337931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.339715 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.339724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.341141 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.341147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.341263 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.341300 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:14:35.341985 (+ 685us) client_negotiation.cc:240] Negotiation successful
0928 21:14:35.342127 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"mutex_wait_us":435,"thread_start_us":109,"threads_started":1}
I20260928 21:14:35.342296 13663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.330108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36515 (local address 127.0.0.1:44092)
0928 21:14:35.334083 (+ 3975us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.334434 (+ 351us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:35.334606 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:35.335307 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:35.335314 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:35.335337 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:35.335487 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.335495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.337784 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.337787 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.339449 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.339460 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.341489 (+ 2029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.341492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.341573 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.341592 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:14:35.342132 (+ 540us) client_negotiation.cc:240] Negotiation successful
0928 21:14:35.342191 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3903,"mutex_wait_us":93,"thread_start_us":95,"threads_started":1}
I20260928 21:14:35.347077 13567 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfd3eb707484fed860cc14f5f803e88 (DEFAULT_TABLE table=test-table [id=10fc9652a5de47669ef24143fd71fd90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:35.348251 13567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfd3eb707484fed860cc14f5f803e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:35.348567 13433 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfd3eb707484fed860cc14f5f803e88 (DEFAULT_TABLE table=test-table [id=10fc9652a5de47669ef24143fd71fd90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:35.349718 13433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfd3eb707484fed860cc14f5f803e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:35.355763 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.330769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37757 (local address 127.0.0.1:57708)
0928 21:14:35.337940 (+ 7171us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.337956 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:35.338074 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:35.338803 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:35.338806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:35.338824 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:35.338925 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.338931 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.343093 (+ 4162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.343097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.353282 (+ 10185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.353298 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.355283 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.355287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.355394 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.355422 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:14:35.355504 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:14:35.355604 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7103,"thread_start_us":82,"threads_started":1}
I20260928 21:14:35.359032 13668 tablet_bootstrap.cc:492] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Bootstrap starting.
I20260928 21:14:35.360510 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.330976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57708 (local address 127.0.98.1:37757)
0928 21:14:35.336349 (+ 5373us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.336358 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.338185 (+ 1827us) server_negotiation.cc:407] Connection header received
0928 21:14:35.338273 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.338279 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.338488 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.338643 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:35.339073 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.340162 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.353998 (+ 13836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.354929 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.355063 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.355212 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.360200 (+ 4988us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.360333 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5244,"thread_start_us":109,"threads_started":1}
I20260928 21:14:35.363622 13287 tablet_service.cc:1467] Processing CreateTablet for tablet 8bfd3eb707484fed860cc14f5f803e88 (DEFAULT_TABLE table=test-table [id=10fc9652a5de47669ef24143fd71fd90]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:35.364773 13287 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bfd3eb707484fed860cc14f5f803e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:35.374588 13669 tablet_bootstrap.cc:492] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Bootstrap starting.
I20260928 21:14:35.375867 13668 tablet_bootstrap.cc:654] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:35.376904 13668 log.cc:824] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:35.380475 13669 tablet_bootstrap.cc:654] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:35.381367 13669 log.cc:824] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:35.384172 13669 tablet_bootstrap.cc:492] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: No bootstrap required, opened a new log
I20260928 21:14:35.384400 13669 ts_tablet_manager.cc:1397] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20260928 21:14:35.386134 13670 tablet_bootstrap.cc:492] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: Bootstrap starting.
I20260928 21:14:35.386391 13668 tablet_bootstrap.cc:492] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: No bootstrap required, opened a new log
I20260928 21:14:35.386659 13668 ts_tablet_manager.cc:1397] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.000s
I20260928 21:14:35.389062 13669 raft_consensus.cc:348] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.389353 13669 raft_consensus.cc:374] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:35.389415 13669 raft_consensus.cc:729] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a35bc90a18b4f378caad4b7e153c9cf, State: Initialized, Role: FOLLOWER
I20260928 21:14:35.389916 13669 consensus_queue.cc:260] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.390960 13669 ts_tablet_manager.cc:1428] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:14:35.391341 13499 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260928 21:14:35.391554 13670 tablet_bootstrap.cc:654] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:35.391489 13668 raft_consensus.cc:348] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.391855 13668 raft_consensus.cc:374] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:35.391943 13668 raft_consensus.cc:729] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ddd9117db324a55affc164bb9332c02, State: Initialized, Role: FOLLOWER
I20260928 21:14:35.392408 13668 consensus_queue.cc:260] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.393474 13668 ts_tablet_manager.cc:1428] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:14:35.393945 13633 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260928 21:14:35.400377 13670 log.cc:824] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:35.403273 13670 tablet_bootstrap.cc:492] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: No bootstrap required, opened a new log
I20260928 21:14:35.403494 13670 ts_tablet_manager.cc:1397] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20260928 21:14:35.407996 13670 raft_consensus.cc:348] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.408291 13670 raft_consensus.cc:374] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:35.408344 13670 raft_consensus.cc:729] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29d27625e448480a8ae1c0c5ea09cb6c, State: Initialized, Role: FOLLOWER
I20260928 21:14:35.408911 13670 consensus_queue.cc:260] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.410008 13670 ts_tablet_manager.cc:1428] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20260928 21:14:35.410382 13365 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
W20260928 21:14:35.460776 13366 tablet.cc:2369] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:14:35.480134 13500 tablet.cc:2369] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:14:35.502328 13634 tablet.cc:2369] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:14:35.664849 13676 raft_consensus.cc:482] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:35.665261 13676 raft_consensus.cc:504] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.666697 13676 leader_election.cc:288] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ddd9117db324a55affc164bb9332c02 (127.0.98.3:44419), 8a35bc90a18b4f378caad4b7e153c9cf (127.0.98.2:36515)
I20260928 21:14:35.671610 13662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.667128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46143 (local address 127.0.98.3:44419)
0928 21:14:35.667323 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.667328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.667435 (+ 107us) server_negotiation.cc:407] Connection header received
0928 21:14:35.667695 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.667698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.667741 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.667833 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:35.668197 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.668983 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.670413 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.670989 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.671094 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.671159 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.671377 (+ 218us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.671475 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260928 21:14:35.671805 13664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.667451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43757 (local address 127.0.98.2:36515)
0928 21:14:35.667567 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.667571 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.667586 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:14:35.667642 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.667645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.667697 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.667795 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:35.668118 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.668960 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.670627 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.671215 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.671339 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.671402 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.671599 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.671680 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:14:35.671821 13685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.666978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36515 (local address 127.0.98.1:43757)
0928 21:14:35.667455 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.667469 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:35.667537 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:35.667819 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:35.667824 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:35.667842 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:35.667998 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.668005 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.669079 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.669083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.670504 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.670515 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.671343 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.671347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.671449 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.671471 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:14:35.671585 (+ 114us) client_negotiation.cc:240] Negotiation successful
0928 21:14:35.671653 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":91,"thread_start_us":121,"threads_started":1}
I20260928 21:14:35.672001 13684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.666911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44419 (local address 127.0.98.1:46143)
0928 21:14:35.667316 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.667332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:35.667526 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:35.667939 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:35.667942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:35.667960 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:35.668079 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.668085 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.669124 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.669135 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.670266 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:35.670281 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.671117 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.671123 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:35.671226 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.671253 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:14:35.671753 (+ 500us) client_negotiation.cc:240] Negotiation successful
0928 21:14:35.671836 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"mutex_wait_us":46,"thread_start_us":136,"threads_started":1}
I20260928 21:14:35.672676 13586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfd3eb707484fed860cc14f5f803e88" candidate_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ddd9117db324a55affc164bb9332c02" is_pre_election: true
I20260928 21:14:35.672899 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfd3eb707484fed860cc14f5f803e88" candidate_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" is_pre_election: true
I20260928 21:14:35.673064 13586 raft_consensus.cc:2454] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d27625e448480a8ae1c0c5ea09cb6c in term 0.
I20260928 21:14:35.673197 13445 raft_consensus.cc:2454] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 29d27625e448480a8ae1c0c5ea09cb6c in term 0.
I20260928 21:14:35.673583 13253 leader_election.cc:302] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d27625e448480a8ae1c0c5ea09cb6c, 7ddd9117db324a55affc164bb9332c02; no voters:
I20260928 21:14:35.673867 13676 raft_consensus.cc:2781] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:14:35.673959 13676 raft_consensus.cc:482] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:14:35.674010 13676 raft_consensus.cc:3037] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:35.675010 13676 raft_consensus.cc:504] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.675349 13676 leader_election.cc:288] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [CANDIDATE]: Term 1 election: Requested vote from peers 7ddd9117db324a55affc164bb9332c02 (127.0.98.3:44419), 8a35bc90a18b4f378caad4b7e153c9cf (127.0.98.2:36515)
I20260928 21:14:35.675626 13586 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfd3eb707484fed860cc14f5f803e88" candidate_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ddd9117db324a55affc164bb9332c02"
I20260928 21:14:35.675709 13445 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bfd3eb707484fed860cc14f5f803e88" candidate_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a35bc90a18b4f378caad4b7e153c9cf"
I20260928 21:14:35.675808 13586 raft_consensus.cc:3037] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:35.675836 13445 raft_consensus.cc:3037] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:35.676898 13586 raft_consensus.cc:2454] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d27625e448480a8ae1c0c5ea09cb6c in term 1.
I20260928 21:14:35.677217 13253 leader_election.cc:302] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29d27625e448480a8ae1c0c5ea09cb6c, 7ddd9117db324a55affc164bb9332c02; no voters:
I20260928 21:14:35.677244 13445 raft_consensus.cc:2454] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 29d27625e448480a8ae1c0c5ea09cb6c in term 1.
I20260928 21:14:35.677527 13676 raft_consensus.cc:2781] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:35.677783 13676 raft_consensus.cc:686] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [term 1 LEADER]: Becoming Leader. State: Replica: 29d27625e448480a8ae1c0c5ea09cb6c, State: Running, Role: LEADER
I20260928 21:14:35.678113 13676 consensus_queue.cc:237] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } }
I20260928 21:14:35.681664 13167 catalog_manager.cc:5495] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c reported cstate change: term changed from 0 to 1, leader changed from <none> to 29d27625e448480a8ae1c0c5ea09cb6c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "29d27625e448480a8ae1c0c5ea09cb6c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37757 } health_report { overall_health: HEALTHY } } }
I20260928 21:14:35.710625 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.705740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57722 (local address 127.0.98.1:37757)
0928 21:14:35.705869 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:35.705874 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:35.705891 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:14:35.706080 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:35.706082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:35.706141 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:35.706244 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:14:35.706560 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.707391 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.708623 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:35.708993 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:35.709092 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:35.709541 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:14:35.710183 (+ 642us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:14:35.710307 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:35.710427 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:14:35.710487 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:14:35.716735 13445 raft_consensus.cc:1261] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 1 FOLLOWER]: Refusing update from remote peer 29d27625e448480a8ae1c0c5ea09cb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:35.716735 13586 raft_consensus.cc:1261] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 1 FOLLOWER]: Refusing update from remote peer 29d27625e448480a8ae1c0c5ea09cb6c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:35.717535 13676 consensus_queue.cc:1035] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ddd9117db324a55affc164bb9332c02" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:35.717747 13686 consensus_queue.cc:1035] T 8bfd3eb707484fed860cc14f5f803e88 P 29d27625e448480a8ae1c0c5ea09cb6c [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a35bc90a18b4f378caad4b7e153c9cf" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 36515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:35.724752 13689 mvcc.cc:205] Tried to move back new op lower bound from 7334420790127898624 to 7334420789980487680. Current Snapshot: MvccSnapshot[applied={T|T < 7334420790127898624}]
I20260928 21:14:35.725332 13690 mvcc.cc:205] Tried to move back new op lower bound from 7334420790127898624 to 7334420789980487680. Current Snapshot: MvccSnapshot[applied={T|T < 7334420790127898624}]
I20260928 21:14:35.737629 13166 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:14:35.737829 13166 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33384:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:14:35.740568 13166 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fe4c66fee4c34ca18cab2a09d8b7918b: Sending DeleteTablet for 3 replicas of tablet 8bfd3eb707484fed860cc14f5f803e88
I20260928 21:14:35.741523 13567 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfd3eb707484fed860cc14f5f803e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39272
I20260928 21:14:35.742079 13699 tablet_replica.cc:330] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: stopping tablet replica
I20260928 21:14:35.742228 13433 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfd3eb707484fed860cc14f5f803e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44092
I20260928 21:14:35.742570 13699 raft_consensus.cc:2229] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:14:35.742924 13699 raft_consensus.cc:2258] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:35.743399 13287 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bfd3eb707484fed860cc14f5f803e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57708
I20260928 21:14:35.743661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13205
I20260928 21:14:35.745249 13699 ts_tablet_manager.cc:1905] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:14:35.748762 13699 ts_tablet_manager.cc:1918] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:14:35.748904 13699 log.cc:1196] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/8bfd3eb707484fed860cc14f5f803e88
I20260928 21:14:35.749239 13699 ts_tablet_manager.cc:1939] T 8bfd3eb707484fed860cc14f5f803e88 P 7ddd9117db324a55affc164bb9332c02: Deleting consensus metadata
I20260928 21:14:35.749454 13700 tablet_replica.cc:330] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: stopping tablet replica
I20260928 21:14:35.749840 13700 raft_consensus.cc:2229] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:14:35.750221 13700 raft_consensus.cc:2258] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:35.750550 13152 catalog_manager.cc:4841] TS 7ddd9117db324a55affc164bb9332c02 (127.0.98.3:44419): tablet 8bfd3eb707484fed860cc14f5f803e88 (table test-table [id=10fc9652a5de47669ef24143fd71fd90]) successfully deleted
I20260928 21:14:35.751694 13700 ts_tablet_manager.cc:1905] T 8bfd3eb707484fed860cc14f5f803e88 P 8a35bc90a18b4f378caad4b7e153c9cf: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:14:35.752148 13151 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37757 (error 108)
I20260928 21:14:35.752432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13369
I20260928 21:14:35.754096 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.753482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37757 (local address 127.0.0.1:57728)
0928 21:14:35.753928 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.754003 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352}
W20260928 21:14:35.755210 13151 catalog_manager.cc:4584] TS 29d27625e448480a8ae1c0c5ea09cb6c (127.0.98.1:37757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bfd3eb707484fed860cc14f5f803e88: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37757: connect: Connection refused (error 111)
I20260928 21:14:35.758208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13502
I20260928 21:14:35.759095 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:35.758407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:36515 (local address 127.0.0.1:44100)
0928 21:14:35.758938 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:35.758996 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:36515: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427}
I20260928 21:14:35.766309 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13134
I20260928 21:14:35.773454 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260928 21:14:36.108353 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260928 21:14:36.108464 392 mini_ranger.cc:66] Stopping Ranger...
I20260928 21:14:36.453732 392 mini_ranger.cc:68] Stopped Ranger
2026-09-28 21:14:36.453 UTC [12201] LOG: received smart shutdown request
2026-09-28 21:14:36.457 UTC [12201] LOG: background worker "logical replication launcher" (PID 12210) exited with exit code 1
2026-09-28 21:14:36.457 UTC [12205] LOG: shutting down
2026-09-28 21:14:36.621 UTC [12201] LOG: database system is shut down
2026-09-28T21:14:36Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48189 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:14:36 dist-test-slave-v7xz krb5kdc[13711](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:14:36 dist-test-slave-v7xz krb5kdc[13711](info): set up 2 sockets
Sep 28 21:14:36 dist-test-slave-v7xz krb5kdc[13711](info): commencing operation
krb5kdc: starting...
W20260928 21:14:37.780997 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:14:37 dist-test-slave-v7xz krb5kdc[13711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:14:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:14:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:14:37.903635 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46327
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46327,127.0.98.61:42841,127.0.98.60:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:38.017009 13724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:38.017313 13724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:38.017356 13724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:38.021587 13724 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:38.021656 13724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:38.021677 13724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:38.021694 13724 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:38.021709 13724 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:38.027163 13724 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46327,127.0.98.61:42841,127.0.98.60:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:38.028362 13724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:38.029567 13724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:38.038208 13729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:38.038231 13730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:38.038208 13732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:38.038625 13724 server_base.cc:1034] running on GCE node
I20260928 21:14:38.039299 13724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:38.040544 13724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:38.041728 13724 hybrid_clock.cc:648] HybridClock initialized: now 1790630078041682 us; error 60 us; skew 500 ppm
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:38.044903 13724 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:38.045866 13724 webserver.cc:460] Webserver started at http://127.0.98.62:45595/ using document root <none> and password file <none>
I20260928 21:14:38.046449 13724 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:38.046517 13724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:38.046698 13724 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:38.049047 13724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "8a376153de074b6497519030e19f101a"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "8b078f03922a0e05781e2598205dd8ca"
server_key_iv: "87561e379018301dc5e27b9b9f34b4c4"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.049698 13724 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "8a376153de074b6497519030e19f101a"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "8b078f03922a0e05781e2598205dd8ca"
server_key_iv: "87561e379018301dc5e27b9b9f34b4c4"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.053902 13724 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:14:38.056664 13739 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:38.058105 13724 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:14:38.058290 13724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "8a376153de074b6497519030e19f101a"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "8b078f03922a0e05781e2598205dd8ca"
server_key_iv: "87561e379018301dc5e27b9b9f34b4c4"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.058424 13724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:38.083534 13724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:38.087594 13724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:38.087811 13724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:38.095135 13724 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46327
I20260928 21:14:38.095211 13791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46327 every 8 connection(s)
I20260928 21:14:38.096150 13724 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:38.099113 13792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:38.100487 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13724
I20260928 21:14:38.100634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:14:38.100976 392 external_mini_cluster.cc:1428] Setting key a12da529b800242f52340fb20a77f2e0
I20260928 21:14:38.104269 13792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:38.120776 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.107057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46327 (local address 127.0.0.1:58682)
0928 21:14:38.107608 (+ 551us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.107666 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.108696 (+ 1030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.109502 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.109508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.109933 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.110259 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.110274 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.111536 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.111539 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.112403 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.112412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.113160 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.113164 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.113269 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.113954 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.113972 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.115471 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.117583 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.117592 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.117604 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.117839 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.118181 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.118184 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.118186 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.118303 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.118920 (+ 617us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.118927 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.119076 (+ 149us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.119256 (+ 180us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.119651 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":111,"spinlock_wait_cycles":1792,"thread_start_us":162,"threads_started":1}
I20260928 21:14:38.120793 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.108218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58682 (local address 127.0.98.62:46327)
0928 21:14:38.108609 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.108622 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.108670 (+ 48us) server_negotiation.cc:407] Connection header received
0928 21:14:38.109054 (+ 384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.109064 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.109309 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.109501 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.110407 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.111376 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.112541 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.113003 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.113177 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.115638 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.115644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.115661 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.115688 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.117449 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.117952 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.117959 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.117961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.118081 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.118394 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.118399 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.118401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.118806 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.118953 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.119316 (+ 363us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.119595 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"spinlock_wait_cycles":1920,"thread_start_us":183,"threads_started":1}
I20260928 21:14:38.123370 13792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:14:38.124244 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.123904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43934)
0928 21:14:38.124033 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.124143 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 21:14:38.124496 13740 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
I20260928 21:14:38.125586 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.125259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43948)
0928 21:14:38.125393 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.125497 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260928 21:14:38.125766 13792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42841: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 21:14:38.129082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42841
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46327,127.0.98.61:42841,127.0.98.60:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:14:38.175046 13792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } attempt: 1
I20260928 21:14:38.176043 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.175700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43952)
0928 21:14:38.175869 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.175935 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260928 21:14:38.176923 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.176637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43956)
0928 21:14:38.176779 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.176855 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 21:14:38.177158 13792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42841: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
W20260928 21:14:38.242625 13799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:38.242918 13799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:38.243007 13799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:14:38.245353 13792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } attempt: 2
I20260928 21:14:38.246268 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.245944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43964)
0928 21:14:38.246131 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.246193 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 21:14:38.247020 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.246801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43972)
0928 21:14:38.246915 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.246965 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:14:38.247149 13792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42841: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42841: connect: Connection refused (error 111)
W20260928 21:14:38.247596 13799 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:38.247639 13799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:38.247677 13799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:38.247717 13799 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:38.247766 13799 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:38.253100 13799 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46327,127.0.98.61:42841,127.0.98.60:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:38.254129 13799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:38.255174 13799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:38.263559 13805 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:38.263550 13807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:38.263547 13804 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:38.263895 13799 server_base.cc:1034] running on GCE node
I20260928 21:14:38.264420 13799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:38.265556 13799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:38.266741 13799 hybrid_clock.cc:648] HybridClock initialized: now 1790630078266728 us; error 35 us; skew 500 ppm
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:38.269577 13799 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 21:14:38.270439 13799 webserver.cc:460] Webserver started at http://127.0.98.61:37883/ using document root <none> and password file <none>
I20260928 21:14:38.271020 13799 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:38.271111 13799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:38.271347 13799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:38.273406 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data/instance:
uuid: "64c735a4326e43ab89735f76d9baf672"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "6324ee17e8c6af833179996e59794f0a"
server_key_iv: "006c67e8760c3946f9fe15d4082d9534"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.273949 13799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal/instance:
uuid: "64c735a4326e43ab89735f76d9baf672"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "6324ee17e8c6af833179996e59794f0a"
server_key_iv: "006c67e8760c3946f9fe15d4082d9534"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.277669 13799 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:14:38.280098 13814 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:38.281428 13799 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:14:38.281569 13799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal
uuid: "64c735a4326e43ab89735f76d9baf672"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "6324ee17e8c6af833179996e59794f0a"
server_key_iv: "006c67e8760c3946f9fe15d4082d9534"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.281677 13799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:38.297832 13799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:38.300624 13799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:38.300848 13799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:38.307898 13799 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42841
I20260928 21:14:38.307928 13866 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42841 every 8 connection(s)
I20260928 21:14:38.308879 13799 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
I20260928 21:14:38.311569 13867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:38.315361 13867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:14:38.315832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13799
I20260928 21:14:38.315997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-1/wal/instance
I20260928 21:14:38.316274 392 external_mini_cluster.cc:1428] Setting key 490ec43dc2ec85a91b53b34473536520
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:38.332204 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.317732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58684 (local address 127.0.98.62:46327)
0928 21:14:38.317878 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.317883 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.318794 (+ 911us) server_negotiation.cc:407] Connection header received
0928 21:14:38.319743 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.319747 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.319811 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.319910 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.321162 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.321691 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.322834 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.323073 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.323204 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.327691 (+ 4487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.327694 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.327696 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.327722 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.329575 (+ 1853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.330146 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.330152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.330157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.330252 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.330826 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.330830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.330832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.331087 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.331199 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.331792 (+ 593us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.331953 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:14:38.333137 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.318106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46327 (local address 127.0.0.1:58684)
0928 21:14:38.318592 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.318645 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.319507 (+ 862us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.320087 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.320094 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.320437 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.321004 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.321020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.321833 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.321836 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.322700 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.322710 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.323256 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.323260 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.323426 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.324106 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.324124 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.327466 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.329719 (+ 2253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.329724 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.329738 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.330033 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.330426 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.330429 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.330430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.330705 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.331206 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.331211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.331515 (+ 304us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.331687 (+ 172us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.331986 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":141,"threads_started":1}
I20260928 21:14:38.334755 13867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:14:38.338321 13792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } attempt: 3
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 21:14:38.343776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41605
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40081
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46327,127.0.98.61:42841,127.0.98.60:41605
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 21:14:38.345758 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.335309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43982)
0928 21:14:38.335463 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.335480 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.335608 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.336188 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.336191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.336379 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.336479 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.336484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.337731 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.337737 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.338336 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.338349 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.339242 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.339245 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.339364 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.340018 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.340037 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.341513 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.343785 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.343788 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.343790 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.344053 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.344377 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.344383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.344388 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.344495 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.345203 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.345206 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.345302 (+ 96us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.345412 (+ 110us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.345544 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 21:14:38.346935 13872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.335440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43982 (local address 127.0.98.61:42841)
0928 21:14:38.335817 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.335827 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.335846 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:14:38.335882 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.335886 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.336041 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.336162 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.336583 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.337569 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.338468 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.339061 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.339173 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.341675 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.341684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.341703 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.341723 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.343665 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.344175 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.344183 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.344189 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.344268 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.344606 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.344609 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.344610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.345007 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.345164 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.346120 (+ 956us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.346817 (+ 697us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":118,"threads_started":1}
I20260928 21:14:38.348517 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.338853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43992 (local address 127.0.98.61:42841)
0928 21:14:38.339187 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.339197 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.339207 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:14:38.339237 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.339240 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.339290 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.339372 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.340274 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.341031 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.342003 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.342241 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.342345 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.344430 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.344433 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.344435 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.344455 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.346699 (+ 2244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.347212 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.347217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.347219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.347304 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.347636 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.347638 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.347640 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.347828 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.347924 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.348260 (+ 336us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.348369 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":173,"threads_started":1}
I20260928 21:14:38.348591 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.338805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:43992)
0928 21:14:38.338939 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.338955 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.339038 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.339401 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.339403 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.339544 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.340160 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.340171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.341275 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.341278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.341861 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.341869 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.342360 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.342363 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.342465 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.342928 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.342948 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.344282 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.346806 (+ 2524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.346813 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.346818 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.347103 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.347420 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.347426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.347431 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.347537 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.347941 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.347946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.348073 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.348197 (+ 124us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.348352 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:14:38.349110 13867 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:14:38.349553 13792 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:14:38.349839 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.349586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60946)
0928 21:14:38.349699 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.349773 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:14:38.350029 13817 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
I20260928 21:14:38.350425 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.350041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60962)
0928 21:14:38.350260 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.350330 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260928 21:14:38.351035 13876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.350653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60966)
0928 21:14:38.350858 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.350907 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":88,"threads_started":1}
I20260928 21:14:38.351332 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.351103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60974)
0928 21:14:38.351224 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.351272 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260928 21:14:38.351297 13867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41605: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
W20260928 21:14:38.351523 13792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41605: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
I20260928 21:14:38.382716 13792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } attempt: 1
I20260928 21:14:38.383596 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.383290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60980)
0928 21:14:38.383435 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.383512 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 21:14:38.384373 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.384147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60984)
0928 21:14:38.384256 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.384312 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:14:38.384522 13792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41605: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
I20260928 21:14:38.400597 13867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } attempt: 1
I20260928 21:14:38.401521 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.401225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60990)
0928 21:14:38.401369 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.401441 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 21:14:38.402287 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.402091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:60994)
0928 21:14:38.402188 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.402234 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260928 21:14:38.402432 13867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41605: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
W20260928 21:14:38.459546 13874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260928 21:14:38.459684 13792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } attempt: 2
W20260928 21:14:38.459867 13874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:38.459944 13874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:14:38.460592 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.460256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32776)
0928 21:14:38.460427 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.460496 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260928 21:14:38.461404 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.461173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32786)
0928 21:14:38.461285 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.461337 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 21:14:38.461609 13792 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41605: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
W20260928 21:14:38.464720 13874 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:38.464777 13874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:38.464798 13874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:38.464815 13874 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:38.464890 13874 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:38.470432 13874 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40081
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46327,127.0.98.61:42841,127.0.98.60:41605
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:41605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:38.470611 13867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } attempt: 2
I20260928 21:14:38.471412 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.471102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32802)
0928 21:14:38.471259 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.471333 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260928 21:14:38.471673 13874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:38.472245 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.472003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32812)
0928 21:14:38.472127 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.472185 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 21:14:38.472391 13867 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:41605: Network error: Client connection negotiation failed: client connection to 127.0.98.60:41605: connect: Connection refused (error 111)
I20260928 21:14:38.472903 13874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:38.481360 13881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:38.481360 13884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:38.481365 13882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:38.481801 13874 server_base.cc:1034] running on GCE node
I20260928 21:14:38.482372 13874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:38.483508 13874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:38.484675 13874 hybrid_clock.cc:648] HybridClock initialized: now 1790630078484662 us; error 25 us; skew 500 ppm
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:38.487645 13874 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 21:14:38.488610 13874 webserver.cc:460] Webserver started at http://127.0.98.60:35579/ using document root <none> and password file <none>
I20260928 21:14:38.489244 13874 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:38.489341 13874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:38.489569 13874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:38.491677 13874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data/instance:
uuid: "8ca4e021eebc4cebb6b56525ed9433da"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "2be848845e4cc15fd82b4112bfa75034"
server_key_iv: "070784dfe461d3ca612d3bdf57d7e771"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.492239 13874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal/instance:
uuid: "8ca4e021eebc4cebb6b56525ed9433da"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "2be848845e4cc15fd82b4112bfa75034"
server_key_iv: "070784dfe461d3ca612d3bdf57d7e771"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.495999 13874 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 21:14:38.498503 13891 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:38.499686 13874 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:14:38.499828 13874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal
uuid: "8ca4e021eebc4cebb6b56525ed9433da"
format_stamp: "Formatted at 2026-09-28 21:14:38 on dist-test-slave-v7xz"
server_key: "2be848845e4cc15fd82b4112bfa75034"
server_key_iv: "070784dfe461d3ca612d3bdf57d7e771"
server_key_version: "encryptionkey@0"
I20260928 21:14:38.499939 13874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:38.513837 13874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:38.516732 13874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:38.516963 13874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:38.524041 13874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:41605
I20260928 21:14:38.524065 13943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:41605 every 8 connection(s)
I20260928 21:14:38.525045 13874 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
I20260928 21:14:38.527830 13944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:38.531373 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 13874
I20260928 21:14:38.531533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790629916120794-392-0/minicluster-data/master-2/wal/instance
I20260928 21:14:38.531811 392 external_mini_cluster.cc:1428] Setting key 01c262ae7466eb75f2016b38958d7a1e
I20260928 21:14:38.531710 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 21:14:38.544725 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.533763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58696 (local address 127.0.98.62:46327)
0928 21:14:38.533907 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.533912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.534222 (+ 310us) server_negotiation.cc:407] Connection header received
0928 21:14:38.534779 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.534781 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.534826 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.534900 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.535869 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.536378 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.537540 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.537796 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.537875 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.540770 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.540774 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.540776 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.540807 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.542603 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.543154 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.543157 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.543159 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.543242 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.543593 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.543598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.543600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.543801 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.543909 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.544426 (+ 517us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.544548 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:14:38.545652 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.533682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46327 (local address 127.0.0.1:58696)
0928 21:14:38.534057 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.534112 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.534556 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.534974 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.534979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.535426 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.535721 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.535733 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.536490 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.536494 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.537416 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.537426 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.537931 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.537935 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.538125 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.538863 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.538879 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.540461 (+ 1582us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.542721 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.542731 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.542742 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.543047 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.543343 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.543347 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.543349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.543498 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.543908 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.543917 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.544199 (+ 282us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.544373 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.544694 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":41,"thread_start_us":131,"threads_started":1}
I20260928 21:14:38.545660 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.533210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32822 (local address 127.0.98.60:41605)
0928 21:14:38.533609 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.533624 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.533688 (+ 64us) server_negotiation.cc:407] Connection header received
0928 21:14:38.534556 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.534579 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.534820 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.535007 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.535721 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.536705 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.537635 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.538069 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.538169 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.541176 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.541181 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.541192 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.541214 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.542975 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.543455 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.543460 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.543461 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.543559 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.543854 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.543858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.543859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.544187 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.544388 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.544630 (+ 242us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.544927 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":33,"server-negotiator.queue_time_us":259,"thread_start_us":132,"threads_started":1}
I20260928 21:14:38.546947 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:38.558305 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.547724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44004 (local address 127.0.98.61:42841)
0928 21:14:38.547832 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.547836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.547873 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:14:38.547918 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.547921 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.547990 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.548070 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.548555 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.549287 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.550169 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.550568 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.550661 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.553321 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.553324 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.553326 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.553353 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.556214 (+ 2861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.556764 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.556768 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.556770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.556925 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.557309 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.557313 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.557315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.557583 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.557645 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.558017 (+ 372us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.558159 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260928 21:14:38.558305 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.547454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42841 (local address 127.0.0.1:44004)
0928 21:14:38.547578 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.547594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.547672 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.548123 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.548127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.548315 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.548435 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.548440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.549387 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.549390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.550047 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.550055 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.550677 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.550681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.550777 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.551496 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.551511 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.553165 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.556347 (+ 3182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.556351 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.556353 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.556645 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.557036 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.557057 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.557060 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.557183 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.557732 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.557735 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.557842 (+ 107us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.557948 (+ 106us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.558124 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:14:38.559512 13944 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:14:38.560798 13792 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } attempt: 3
I20260928 21:14:38.560964 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.547862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.0.98.62:46327)
0928 21:14:38.547964 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.547974 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.548011 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:14:38.548144 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.548147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.548196 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.548280 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.548975 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.549544 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.550566 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.550874 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.550985 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.554192 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.554194 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.554197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.554238 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.558258 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.558847 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.558854 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.558858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.558950 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.559421 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.559425 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.559426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.560094 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.560317 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.560514 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.560673 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260928 21:14:38.563532 13867 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } attempt: 3
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:14:38 dist-test-slave-v7xz krb5kdc[13711](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630078, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 21:14:38.576709 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.564685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44018 (local address 127.0.98.61:42841)
0928 21:14:38.564905 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.564909 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.564922 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:14:38.564956 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.564959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.565000 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.565085 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.566010 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.566809 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.567898 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.568234 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.568330 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.571051 (+ 2721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.571056 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.571063 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.571096 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.574810 (+ 3714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.575367 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.575372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.575376 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.575468 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.575796 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.575799 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.575800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.576034 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.576283 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.576396 (+ 113us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.576512 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":104}
I20260928 21:14:38.577564 13795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.561416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32840)
0928 21:14:38.561589 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.561605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.561675 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.561886 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.561889 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.562106 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.562221 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.562232 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.563469 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.563472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.564107 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.564115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.564842 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.564846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.564964 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.565625 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.565642 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.567174 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.571038 (+ 3864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.571042 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.571047 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.571307 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.571664 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.571669 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.571673 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.571766 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.572251 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.572254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.577111 (+ 4857us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.577222 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.577379 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87}
I20260928 21:14:38.577625 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.561535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32840 (local address 127.0.98.60:41605)
0928 21:14:38.561667 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.561671 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.561685 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:14:38.561760 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.561763 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.561810 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.561878 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.562356 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.563356 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.564227 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.564662 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.564774 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.567318 (+ 2544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.567322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.567324 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.567347 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.570901 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.571453 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.571457 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.571460 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.571551 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.571879 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.571882 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.571884 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.572138 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.572258 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.577316 (+ 5058us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.577442 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56,"spinlock_wait_cycles":5760}
I20260928 21:14:38.577705 13870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.564379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32856)
0928 21:14:38.564478 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.564499 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.564570 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.566340 (+ 1770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.566344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.566520 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.566620 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.566627 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.567786 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.567791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.568522 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.568529 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.569172 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.569176 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.569258 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.570501 (+ 1243us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.570519 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.572922 (+ 2403us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.575940 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.575944 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.575947 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.576228 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.576571 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.576577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.576581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.576674 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.577251 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.577254 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.577347 (+ 93us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.577427 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.577553 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:14:38.578251 13950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.565038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32856 (local address 127.0.98.60:41605)
0928 21:14:38.565427 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.565439 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.565451 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:14:38.565519 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.565524 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.566193 (+ 669us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.566733 (+ 540us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.566864 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.567678 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.568633 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.569043 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.569174 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.573231 (+ 4057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.573235 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.573237 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.573262 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.575827 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.576362 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.576367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.576371 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.576463 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.576792 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.576795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.576797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.577122 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.577864 (+ 742us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.577945 (+ 81us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.578078 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":102,"server-negotiator.queue_time_us":306,"thread_start_us":117,"threads_started":1}
I20260928 21:14:38.581255 13948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.560054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:41605 (local address 127.0.0.1:32832)
0928 21:14:38.560218 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:38.560233 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:38.560299 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:38.565966 (+ 5667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:38.565973 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:38.566149 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:38.566230 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.566237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.567188 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.567191 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.568003 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:38.568013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.568713 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.568719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:38.568819 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.570202 (+ 1383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:38.570222 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:38.571736 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:38.579616 (+ 7880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.579619 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.579621 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.579872 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.580192 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:38.580194 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:38.580196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:38.580276 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:38.580766 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:38.580769 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:38.580890 (+ 121us) client_negotiation.cc:769] Sending connection context
0928 21:14:38.580982 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:14:38.581105 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":3}
I20260928 21:14:38.581691 13949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.564808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32832 (local address 127.0.98.60:41605)
0928 21:14:38.565170 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.565185 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.565203 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:14:38.565243 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.565245 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.565294 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.565378 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.566343 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.567079 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.568117 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.568477 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.568592 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.571875 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.571878 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.571880 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.571912 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.579489 (+ 7577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.579983 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.579986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.579988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.580087 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.580363 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.580365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.580367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.580580 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.580680 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.581424 (+ 744us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.581530 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"spinlock_wait_cycles":2816,"thread_start_us":110,"threads_started":1}
I20260928 21:14:38.581679 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: Bootstrap starting.
I20260928 21:14:38.582729 13867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: Bootstrap starting.
I20260928 21:14:38.584045 13792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:38.585449 13867 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:38.585589 13792 log.cc:824] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:38.586328 13867 log.cc:824] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:38.588214 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: Bootstrap starting.
I20260928 21:14:38.588728 13792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: No bootstrap required, opened a new log
I20260928 21:14:38.591063 13944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:38.591908 13867 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: No bootstrap required, opened a new log
I20260928 21:14:38.591923 13944 log.cc:824] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:38.593102 13792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.593364 13792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:38.593426 13792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a376153de074b6497519030e19f101a, State: Initialized, Role: FOLLOWER
I20260928 21:14:38.593928 13792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.594054 13944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: No bootstrap required, opened a new log
I20260928 21:14:38.595408 13792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:38.596007 13867 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.596241 13867 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:38.596297 13867 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64c735a4326e43ab89735f76d9baf672, State: Initialized, Role: FOLLOWER
I20260928 21:14:38.596817 13867 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.596997 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:14:38.598340 13867 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:38.598767 13944 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.599071 13944 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:38.599128 13944 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ca4e021eebc4cebb6b56525ed9433da, State: Initialized, Role: FOLLOWER
I20260928 21:14:38.599638 13944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.601657 13944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:38.605265 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.605392 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:14:38.609357 13996 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:14:38.609462 13996 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:14:38.609752 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.609891 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:14:38.611122 13982 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:14:38.611198 13982 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:14:38.611708 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.611851 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:14:38.614823 13874 master_runner.cc:263] No leader master found from master 8ca4e021eebc4cebb6b56525ed9433da
I20260928 21:14:38.614897 13874 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:14:38.600661 13724 master_runner.cc:186] Error getting master registration for 127.0.98.61:42841: OK, instance_id { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" instance_seqno: 1790630078305754 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:14:38.616572 13724 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:14:38.616928 13799 master_runner.cc:263] No leader master found from master 64c735a4326e43ab89735f76d9baf672
I20260928 21:14:38.617007 13799 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:14:38.618287 13947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.602538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32864 (local address 127.0.98.60:41605)
0928 21:14:38.603010 (+ 472us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.603015 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.603975 (+ 960us) server_negotiation.cc:407] Connection header received
0928 21:14:38.604161 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.604167 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.604226 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.604280 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.605276 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.605948 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.607138 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.607500 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.607572 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.612823 (+ 5251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.612848 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.612851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.612879 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.616123 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.616804 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.616808 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.616810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.616928 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.617382 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.617385 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.617387 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.617576 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.617726 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.617980 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.618109 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385}
W20260928 21:14:38.618288 13966 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:14:38.618369 13966 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:14:38.621305 13873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.601645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44034 (local address 127.0.98.61:42841)
0928 21:14:38.601774 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.601779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.602364 (+ 585us) server_negotiation.cc:407] Connection header received
0928 21:14:38.602649 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.602655 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.602712 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.602804 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.604709 (+ 1905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.605484 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.606786 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.607081 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.607157 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.613680 (+ 6523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.613684 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.613686 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.613711 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.619475 (+ 5764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.619980 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.619984 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.619985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.620082 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.620359 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.620363 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.620365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.620598 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.620745 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.621009 (+ 264us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.621134 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:14:38.621831 13796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:38.603469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.0.98.62:46327)
0928 21:14:38.603579 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:38.603583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:38.605671 (+ 2088us) server_negotiation.cc:407] Connection header received
0928 21:14:38.605829 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:38.605832 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:38.605878 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:38.605927 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:38.611579 (+ 5652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.612359 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.613659 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:38.614024 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:38.614124 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:38.617896 (+ 3772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:38.617903 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:38.617909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:38.617941 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:38.620009 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.620459 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.620462 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.620464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.620538 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:38.620859 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:38.620862 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:38.620864 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:38.621078 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:38.621210 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:38.621554 (+ 344us) server_negotiation.cc:299] Negotiation successful
0928 21:14:38.621688 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260928 21:14:38.768349 13897 tablet.cc:2369] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:14:38.802413 13820 tablet.cc:2369] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:14:38.809819 13976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:38.810038 13976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.811033 13976 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8a376153de074b6497519030e19f101a (127.0.98.62:46327), 64c735a4326e43ab89735f76d9baf672 (127.0.98.61:42841)
I20260928 21:14:38.811587 13766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ca4e021eebc4cebb6b56525ed9433da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a376153de074b6497519030e19f101a" is_pre_election: true
I20260928 21:14:38.811586 13841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ca4e021eebc4cebb6b56525ed9433da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64c735a4326e43ab89735f76d9baf672" is_pre_election: true
I20260928 21:14:38.811910 13766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca4e021eebc4cebb6b56525ed9433da in term 0.
I20260928 21:14:38.811910 13841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ca4e021eebc4cebb6b56525ed9433da in term 0.
I20260928 21:14:38.812338 13892 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 64c735a4326e43ab89735f76d9baf672, 8ca4e021eebc4cebb6b56525ed9433da; no voters:
I20260928 21:14:38.812606 13976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:14:38.812732 13976 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:14:38.812788 13976 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:38.813879 13976 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.814245 13976 leader_election.cc:288] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [CANDIDATE]: Term 1 election: Requested vote from peers 8a376153de074b6497519030e19f101a (127.0.98.62:46327), 64c735a4326e43ab89735f76d9baf672 (127.0.98.61:42841)
I20260928 21:14:38.814597 13766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ca4e021eebc4cebb6b56525ed9433da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8a376153de074b6497519030e19f101a"
I20260928 21:14:38.814744 13766 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:38.815829 13766 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca4e021eebc4cebb6b56525ed9433da in term 1.
I20260928 21:14:38.816200 13895 leader_election.cc:302] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a376153de074b6497519030e19f101a, 8ca4e021eebc4cebb6b56525ed9433da; no voters:
I20260928 21:14:38.816314 13841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8ca4e021eebc4cebb6b56525ed9433da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "64c735a4326e43ab89735f76d9baf672"
I20260928 21:14:38.816376 13976 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:38.816450 13841 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:38.816596 13976 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [term 1 LEADER]: Becoming Leader. State: Replica: 8ca4e021eebc4cebb6b56525ed9433da, State: Running, Role: LEADER
I20260928 21:14:38.816967 13976 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } }
I20260928 21:14:38.817723 13841 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ca4e021eebc4cebb6b56525ed9433da in term 1.
I20260928 21:14:38.818805 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca4e021eebc4cebb6b56525ed9433da. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.818929 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:38.819322 14002 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:38.820349 14002 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:38.826231 13841 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [term 1 FOLLOWER]: Refusing update from remote peer 8ca4e021eebc4cebb6b56525ed9433da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:38.826357 13766 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [term 1 FOLLOWER]: Refusing update from remote peer 8ca4e021eebc4cebb6b56525ed9433da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:38.826948 14000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [LEADER]: Connected to new peer: Peer: permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:38.827482 14000 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [LEADER]: Connected to new peer: Peer: permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:38.832175 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca4e021eebc4cebb6b56525ed9433da. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.832465 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:14:38.834667 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ca4e021eebc4cebb6b56525ed9433da. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.834802 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:14:38.838044 13976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.838078 14000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.838178 13976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:38.838210 14000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:38.839254 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.839381 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:14:38.839520 13954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8ca4e021eebc4cebb6b56525ed9433da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8a376153de074b6497519030e19f101a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46327 } } peers { permanent_uuid: "64c735a4326e43ab89735f76d9baf672" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42841 } } peers { permanent_uuid: "8ca4e021eebc4cebb6b56525ed9433da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 41605 } } }
I20260928 21:14:38.839622 13954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:14:38.842414 14006 mvcc.cc:205] Tried to move back new op lower bound from 7334420802865762304 to 7334420802837544960. Current Snapshot: MvccSnapshot[applied={T|T < 7334420802865762304 or (T in {7334420802865762304})}]
I20260928 21:14:38.842592 14004 mvcc.cc:205] Tried to move back new op lower bound from 7334420802865762304 to 7334420802837544960. Current Snapshot: MvccSnapshot[applied={T|T < 7334420802865762304 or (T in {7334420802865762304})}]
I20260928 21:14:38.842780 14002 catalog_manager.cc:1347] Generated new cluster ID: 22417db0a83e44c595b73c07ad68dd7b
I20260928 21:14:38.842852 14002 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:14:38.860996 14002 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:14:38.862288 14002 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:14:38.875480 14002 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ca4e021eebc4cebb6b56525ed9433da: Generated new TSK 0
I20260928 21:14:38.876253 14002 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:14:39.612419 13982 catalog_manager.cc:1259] Loaded cluster ID: 22417db0a83e44c595b73c07ad68dd7b
I20260928 21:14:39.612540 13982 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: loading cluster ID for follower catalog manager: success
I20260928 21:14:39.614225 13982 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: acquiring CA information for follower catalog manager: success
I20260928 21:14:39.615430 13982 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 64c735a4326e43ab89735f76d9baf672: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:14:39.619479 13966 catalog_manager.cc:1259] Loaded cluster ID: 22417db0a83e44c595b73c07ad68dd7b
I20260928 21:14:39.619587 13966 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: loading cluster ID for follower catalog manager: success
I20260928 21:14:39.621206 13966 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: acquiring CA information for follower catalog manager: success
I20260928 21:14:39.622418 13966 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8a376153de074b6497519030e19f101a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:14:39.939596 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13724
I20260928 21:14:39.946146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13799
I20260928 21:14:39.952296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 13874
2026-09-28T21:14:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3214 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:14:40 dist-test-slave-v7xz krb5kdc[14017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:14:40 dist-test-slave-v7xz krb5kdc[14017](info): set up 2 sockets
Sep 28 21:14:40 dist-test-slave-v7xz krb5kdc[14017](info): commencing operation
krb5kdc: starting...
W20260928 21:14:41.021154 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:14:41 dist-test-slave-v7xz krb5kdc[14017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:14:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:14:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:14:41.146169 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42093
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:41.258262 14030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:41.258622 14030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:41.258708 14030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:41.263003 14030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:41.263096 14030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:41.263131 14030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:41.263170 14030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:41.263199 14030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:41.269148 14030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42093
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:41.270306 14030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:41.271374 14030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:41.280155 14038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:41.280109 14036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:41.280159 14035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:41.280444 14030 server_base.cc:1034] running on GCE node
I20260928 21:14:41.281085 14030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:41.282295 14030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:41.283458 14030 hybrid_clock.cc:648] HybridClock initialized: now 1790630081283423 us; error 53 us; skew 500 ppm
Sep 28 21:14:41 dist-test-slave-v7xz krb5kdc[14017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:41.286546 14030 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:41.287585 14030 webserver.cc:460] Webserver started at http://127.0.98.62:42525/ using document root <none> and password file <none>
I20260928 21:14:41.288187 14030 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:41.288281 14030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:41.288520 14030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:41.290665 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "4a007cdb952e4510a0384dd98de64cef"
format_stamp: "Formatted at 2026-09-28 21:14:41 on dist-test-slave-v7xz"
server_key: "50f2431b7a2971735bcda00fe1a49ca5"
server_key_iv: "d7d634437f8c260c4904875f8692c058"
server_key_version: "encryptionkey@0"
I20260928 21:14:41.291253 14030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "4a007cdb952e4510a0384dd98de64cef"
format_stamp: "Formatted at 2026-09-28 21:14:41 on dist-test-slave-v7xz"
server_key: "50f2431b7a2971735bcda00fe1a49ca5"
server_key_iv: "d7d634437f8c260c4904875f8692c058"
server_key_version: "encryptionkey@0"
I20260928 21:14:41.295320 14030 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:14:41.297878 14045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:41.299126 14030 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:14:41.299285 14030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "4a007cdb952e4510a0384dd98de64cef"
format_stamp: "Formatted at 2026-09-28 21:14:41 on dist-test-slave-v7xz"
server_key: "50f2431b7a2971735bcda00fe1a49ca5"
server_key_iv: "d7d634437f8c260c4904875f8692c058"
server_key_version: "encryptionkey@0"
I20260928 21:14:41.299413 14030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:41.312234 14030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:41.315786 14030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:41.316010 14030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:41.323693 14030 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42093
I20260928 21:14:41.323771 14097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42093 every 8 connection(s)
I20260928 21:14:41.324783 14030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:41.327711 14098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:41.332700 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14030
I20260928 21:14:41.332885 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:14:41.333159 392 external_mini_cluster.cc:1428] Setting key 7ad8693150035b5971e78a25cb8eb68f
I20260928 21:14:41.333287 14098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef: Bootstrap starting.
I20260928 21:14:41.335789 14098 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:41.336727 14098 log.cc:824] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:41.338815 14098 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef: No bootstrap required, opened a new log
Sep 28 21:14:41 dist-test-slave-v7xz krb5kdc[14017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630081, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:41.343595 14098 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a007cdb952e4510a0384dd98de64cef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260928 21:14:41.343928 14098 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:41.344007 14098 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a007cdb952e4510a0384dd98de64cef, State: Initialized, Role: FOLLOWER
I20260928 21:14:41.344555 14098 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a007cdb952e4510a0384dd98de64cef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260928 21:14:41.344730 14098 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:14:41.344806 14098 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:14:41.344950 14098 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:41.347743 14098 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a007cdb952e4510a0384dd98de64cef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260928 21:14:41.348119 14098 leader_election.cc:302] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a007cdb952e4510a0384dd98de64cef; no voters:
I20260928 21:14:41.348328 14101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:41.334441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47162 (local address 127.0.98.62:42093)
0928 21:14:41.334855 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:41.334871 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:41.334908 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:14:41.335519 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:41.335536 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:41.335880 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:41.336219 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:41.337165 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:41.338250 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:41.339386 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:41.339847 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:41.339971 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:41.342782 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:41.342790 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:41.342809 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:41.342845 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:41.344928 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:41.345528 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:41.345540 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:41.345546 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:41.345698 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:41.345977 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:41.345981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:41.345984 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:41.346569 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:41.346724 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:41.346965 (+ 241us) server_negotiation.cc:299] Negotiation successful
0928 21:14:41.347243 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":137,"threads_started":1}
I20260928 21:14:41.348479 14098 leader_election.cc:288] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:14:41.348739 14103 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:41.348997 14103 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [term 1 LEADER]: Becoming Leader. State: Replica: 4a007cdb952e4510a0384dd98de64cef, State: Running, Role: LEADER
I20260928 21:14:41.349306 14103 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a007cdb952e4510a0384dd98de64cef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } }
I20260928 21:14:41.349885 14098 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:41.351706 14105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a007cdb952e4510a0384dd98de64cef. Latest consensus state: current_term: 1 leader_uuid: "4a007cdb952e4510a0384dd98de64cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a007cdb952e4510a0384dd98de64cef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } } }
I20260928 21:14:41.351944 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:14:41.353001 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a007cdb952e4510a0384dd98de64cef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a007cdb952e4510a0384dd98de64cef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42093 } } }
I20260928 21:14:41.353340 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:41.353132 14105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef [sys.catalog]: This master's current role is: LEADER
W20260928 21:14:41.355250 14117 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:14:41.355319 14117 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:14:41.355402 14119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:41.356108 14119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:41.361871 14119 catalog_manager.cc:1347] Generated new cluster ID: 4c0e292ec44d4ca5aead264293b4930e
I20260928 21:14:41.361958 14119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:14:41.373472 14119 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:14:41.374508 14119 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:14:41.378476 14119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a007cdb952e4510a0384dd98de64cef: Generated new TSK 0
I20260928 21:14:41.379251 14119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:14:41.408454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14030
2026-09-28T21:14:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1453 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:14:41 dist-test-slave-v7xz krb5kdc[14125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:14:41 dist-test-slave-v7xz krb5kdc[14125](info): set up 2 sockets
Sep 28 21:14:41 dist-test-slave-v7xz krb5kdc[14125](info): commencing operation
krb5kdc: starting...
W20260928 21:14:42.569196 392 mini_kdc.cc:121] Time spent starting KDC: real 1.141s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:14:42 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:14:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:14:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:14:42.692162 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:42.803634 14138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:42.803941 14138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:42.804044 14138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:42.808175 14138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:42.808255 14138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:42.808292 14138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:42.808348 14138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:42.808385 14138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:42.813717 14138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:42.814769 14138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:42.815774 14138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:42.824471 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:42.824502 14146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:42.824484 14138 server_base.cc:1034] running on GCE node
W20260928 21:14:42.824482 14144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:42.825309 14138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:42.826464 14138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:42.827672 14138 hybrid_clock.cc:648] HybridClock initialized: now 1790630082827653 us; error 31 us; skew 500 ppm
Sep 28 21:14:42 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:42.835662 14138 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:42.836730 14138 webserver.cc:460] Webserver started at http://127.0.98.62:35599/ using document root <none> and password file <none>
I20260928 21:14:42.837364 14138 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:42.837464 14138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:42.837707 14138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:42.839799 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "af4c699c797649ea855b52b81c77c3b1"
format_stamp: "Formatted at 2026-09-28 21:14:42 on dist-test-slave-v7xz"
server_key: "207b881ba976b69e12f937e834f1edf1"
server_key_iv: "ff434ac2a4567d21c58485547d5d3797"
server_key_version: "encryptionkey@0"
I20260928 21:14:42.840354 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "af4c699c797649ea855b52b81c77c3b1"
format_stamp: "Formatted at 2026-09-28 21:14:42 on dist-test-slave-v7xz"
server_key: "207b881ba976b69e12f937e834f1edf1"
server_key_iv: "ff434ac2a4567d21c58485547d5d3797"
server_key_version: "encryptionkey@0"
I20260928 21:14:42.844045 14138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:14:42.846966 14153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:42.848641 14138 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:42.848776 14138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "af4c699c797649ea855b52b81c77c3b1"
format_stamp: "Formatted at 2026-09-28 21:14:42 on dist-test-slave-v7xz"
server_key: "207b881ba976b69e12f937e834f1edf1"
server_key_iv: "ff434ac2a4567d21c58485547d5d3797"
server_key_version: "encryptionkey@0"
I20260928 21:14:42.848953 14138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:42.870719 14138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:42.873793 14138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:42.874002 14138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:42.880924 14138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40065
I20260928 21:14:42.881208 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40065 every 8 connection(s)
I20260928 21:14:42.881915 14138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:42.884658 14206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:42.888432 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14138
I20260928 21:14:42.888548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:14:42.888900 392 external_mini_cluster.cc:1428] Setting key 0a51a231835c9cb438d31dc21edbc7db
I20260928 21:14:42.890679 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap starting.
I20260928 21:14:42.893224 14206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:42.893975 14206 log.cc:824] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:42.896138 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: No bootstrap required, opened a new log
Sep 28 21:14:42 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:42.900421 14206 raft_consensus.cc:348] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:42.900672 14206 raft_consensus.cc:374] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:42.900749 14206 raft_consensus.cc:729] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af4c699c797649ea855b52b81c77c3b1, State: Initialized, Role: FOLLOWER
I20260928 21:14:42.901257 14206 consensus_queue.cc:260] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:42.901410 14206 raft_consensus.cc:388] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:14:42.901499 14206 raft_consensus.cc:482] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:14:42.901597 14206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:42.902542 14206 raft_consensus.cc:504] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:42.902933 14206 leader_election.cc:302] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af4c699c797649ea855b52b81c77c3b1; no voters:
I20260928 21:14:42.903281 14206 leader_election.cc:288] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:14:42.903353 14211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:42.903553 14211 raft_consensus.cc:686] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 LEADER]: Becoming Leader. State: Replica: af4c699c797649ea855b52b81c77c3b1, State: Running, Role: LEADER
I20260928 21:14:42.903787 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:42.890372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34356 (local address 127.0.98.62:40065)
0928 21:14:42.890790 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:42.890813 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:42.890857 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:14:42.891479 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:42.891495 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:42.891825 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:42.892103 (+ 278us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:42.893057 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:42.894142 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:42.895283 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:42.895757 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:42.895895 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:42.898680 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:42.898685 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:42.898698 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:42.898725 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:42.900529 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:42.901121 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:42.901127 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:42.901131 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:42.901247 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:42.901575 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:42.901580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:42.901582 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:42.902013 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:42.902245 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:42.902489 (+ 244us) server_negotiation.cc:299] Negotiation successful
0928 21:14:42.902813 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":151,"threads_started":1}
I20260928 21:14:42.903890 14211 consensus_queue.cc:237] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:42.904567 14206 sys_catalog.cc:564] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:42.905824 14212 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af4c699c797649ea855b52b81c77c3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } }
I20260928 21:14:42.905951 14212 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:42.906388 14213 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af4c699c797649ea855b52b81c77c3b1. Latest consensus state: current_term: 1 leader_uuid: "af4c699c797649ea855b52b81c77c3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } }
I20260928 21:14:42.906548 14213 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:42.906642 14220 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:42.909570 14220 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:42.915580 14220 catalog_manager.cc:1347] Generated new cluster ID: 6e511739f3de4faaa1c28b13217d4dfa
I20260928 21:14:42.915694 14220 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:14:42.935316 14220 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:14:42.936357 14220 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:14:42.943375 14220 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Generated new TSK 0
I20260928 21:14:42.944168 14220 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:14:42.979875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40065
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:43.108374 14232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:43.108718 14232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:43.108796 14232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:43.112947 14232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:43.113058 14232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:43.113157 14232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:14:43.118673 14232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:43.119848 14232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:43.120966 14232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:43.130642 14238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:43.130643 14237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:43.130669 14240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:43.130909 14232 server_base.cc:1034] running on GCE node
I20260928 21:14:43.131389 14232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:43.132066 14232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:43.133253 14232 hybrid_clock.cc:648] HybridClock initialized: now 1790630083133239 us; error 33 us; skew 500 ppm
Sep 28 21:14:43 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:43.135980 14232 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:14:43.137035 14232 webserver.cc:460] Webserver started at http://127.0.98.1:37321/ using document root <none> and password file <none>
I20260928 21:14:43.137601 14232 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:43.137694 14232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:43.137930 14232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:43.140005 14232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "d895b7a9199646e8a73a54a6559e9402"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "bbbd5d55adb9a6f3153b11a1e37be3a3"
server_key_iv: "db6e8f68ca2645caf8b127b607067901"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.140582 14232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d895b7a9199646e8a73a54a6559e9402"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "bbbd5d55adb9a6f3153b11a1e37be3a3"
server_key_iv: "db6e8f68ca2645caf8b127b607067901"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.144308 14232 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:43.146938 14247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.148186 14232 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:14:43.148332 14232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "d895b7a9199646e8a73a54a6559e9402"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "bbbd5d55adb9a6f3153b11a1e37be3a3"
server_key_iv: "db6e8f68ca2645caf8b127b607067901"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.148455 14232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:43.186334 14232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:43.190073 14232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:43.190411 14232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:43.191130 14232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:43.192026 14232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:43.192101 14232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.192173 14232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:43.192215 14232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.201440 14232 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36377
I20260928 21:14:43.201457 14360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36377 every 8 connection(s)
I20260928 21:14:43.202404 14232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:14:43.207832 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14232
I20260928 21:14:43.207952 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:14:43.208251 392 external_mini_cluster.cc:1428] Setting key 9197777f87938cd93f113b8bc951c989
Sep 28 21:14:43 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:14:43.217989 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.204441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46175 (local address 127.0.98.62:40065)
0928 21:14:43.204583 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.204587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.205768 (+ 1181us) server_negotiation.cc:407] Connection header received
0928 21:14:43.207145 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.207149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.207200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.207293 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:43.208651 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.209337 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.210625 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.211060 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.211186 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.214195 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:43.214199 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:43.214201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:43.214226 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:43.215825 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.216382 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.216387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.216389 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.216481 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.216867 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.216871 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.216872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.217058 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:43.217177 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.217681 (+ 504us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.217813 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:14:43.218918 14363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.204998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:46175)
0928 21:14:43.205596 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.205644 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.206908 (+ 1264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.207459 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.207466 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.207872 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:43.208488 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.208504 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.209535 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.209539 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.210491 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.210501 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.211280 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.211286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.211530 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.212239 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:43.212268 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:43.213999 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:43.215959 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:43.215965 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:43.215982 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:43.216276 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:43.216583 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:43.216587 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:43.216590 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:43.216737 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:43.217177 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:43.217186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:43.217435 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.217628 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.217959 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":169,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:14:43.220367 14361 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40065
I20260928 21:14:43.220746 14361 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:43.221462 14361 heartbeater.cc:511] Master 127.0.98.62:40065 requested a full tablet report, sending...
I20260928 21:14:43.223268 14170 ts_manager.cc:194] Registered new tserver with Master: d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377)
I20260928 21:14:43.224607 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46175
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:14:43.235822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40065
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:43.346966 14366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:43.347267 14366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:43.347314 14366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:43.351449 14366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:43.351514 14366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:43.351574 14366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:14:43.357410 14366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:43.358531 14366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:43.359593 14366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:43.369437 14374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:43.369449 14372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:43.369463 14371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:43.369762 14366 server_base.cc:1034] running on GCE node
I20260928 21:14:43.370323 14366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:43.370999 14366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:43.372181 14366 hybrid_clock.cc:648] HybridClock initialized: now 1790630083372148 us; error 47 us; skew 500 ppm
Sep 28 21:14:43 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:43.375007 14366 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:14:43.375888 14366 webserver.cc:460] Webserver started at http://127.0.98.2:36791/ using document root <none> and password file <none>
I20260928 21:14:43.376459 14366 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:43.376550 14366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:43.376792 14366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:43.378836 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "0c8825d2f0d84d919c4496ca4f013b74"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "3384c29610fe84e7feaad23d81a5aa46"
server_key_iv: "c0d3ef3cacca8bf52fa98a6b45f49e11"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.379384 14366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0c8825d2f0d84d919c4496ca4f013b74"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "3384c29610fe84e7feaad23d81a5aa46"
server_key_iv: "c0d3ef3cacca8bf52fa98a6b45f49e11"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.383087 14366 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260928 21:14:43.385403 14381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.386561 14366 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:14:43.386698 14366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "0c8825d2f0d84d919c4496ca4f013b74"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "3384c29610fe84e7feaad23d81a5aa46"
server_key_iv: "c0d3ef3cacca8bf52fa98a6b45f49e11"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.386817 14366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:43.398406 14366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:43.401871 14366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:43.402076 14366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:43.402688 14366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:43.403514 14366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:43.403585 14366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.403674 14366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:43.403729 14366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.413085 14366 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37111
I20260928 21:14:43.413112 14494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37111 every 8 connection(s)
I20260928 21:14:43.413970 14366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:14:43.422109 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14366
I20260928 21:14:43.422256 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:14:43.422605 392 external_mini_cluster.cc:1428] Setting key 19aee8bc3ad4aecdd480f817ab8f806c
Sep 28 21:14:43 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:43.428928 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.416003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56983 (local address 127.0.98.62:40065)
0928 21:14:43.416166 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.416169 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.417093 (+ 924us) server_negotiation.cc:407] Connection header received
0928 21:14:43.418420 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.418423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.418481 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.418590 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:43.419784 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.420323 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.421559 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.421789 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.421906 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.424696 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:43.424699 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:43.424701 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:43.424725 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:43.426583 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.427094 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.427100 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.427105 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.427195 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.427570 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.427576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.427581 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.427853 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:43.427962 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.428540 (+ 578us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.428694 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260928 21:14:43.429706 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.416427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:56983)
0928 21:14:43.416939 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.416982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.418193 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.418763 (+ 570us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.418770 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.419144 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:43.419622 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.419635 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.420455 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.420463 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.421421 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.421430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.421946 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.421950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.422124 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.422811 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:43.422828 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:43.424520 (+ 1692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:43.426707 (+ 2187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:43.426712 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:43.426727 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:43.426985 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:43.427290 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:43.427292 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:43.427294 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:43.427428 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:43.427959 (+ 531us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:43.427964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:43.428234 (+ 270us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.428457 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.428728 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260928 21:14:43.430922 14495 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40065
I20260928 21:14:43.431173 14495 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:43.431667 14495 heartbeater.cc:511] Master 127.0.98.62:40065 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:14:43.432739 14170 ts_manager.cc:194] Registered new tserver with Master: 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
I20260928 21:14:43.433411 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56983
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:14:43.450178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40065
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:43.562963 14500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:43.563242 14500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:43.563319 14500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:43.567745 14500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:43.567811 14500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:43.567904 14500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:14:43.573477 14500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:43.574558 14500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:43.575598 14500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:43.585450 14508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:43.585454 14505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:43.585451 14506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:43.585511 14500 server_base.cc:1034] running on GCE node
I20260928 21:14:43.586213 14500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:43.586879 14500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:43.588053 14500 hybrid_clock.cc:648] HybridClock initialized: now 1790630083588038 us; error 30 us; skew 500 ppm
Sep 28 21:14:43 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:43.591055 14500 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:14:43.591986 14500 webserver.cc:460] Webserver started at http://127.0.98.3:36865/ using document root <none> and password file <none>
I20260928 21:14:43.592559 14500 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:43.592651 14500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:43.592945 14500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:14:43.595021 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "b2a9cb6252460ec78032998be15891c7"
server_key_iv: "c173984acd3707d400c6be6c12833513"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.595834 14500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "b2a9cb6252460ec78032998be15891c7"
server_key_iv: "c173984acd3707d400c6be6c12833513"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.600028 14500 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.005s
I20260928 21:14:43.602671 14515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.603868 14500 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:14:43.604009 14500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "b2a9cb6252460ec78032998be15891c7"
server_key_iv: "c173984acd3707d400c6be6c12833513"
server_key_version: "encryptionkey@0"
I20260928 21:14:43.604203 14500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:43.617662 14500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:43.621328 14500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:43.621564 14500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:43.622197 14500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:43.623044 14500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:43.623091 14500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.623131 14500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:43.623147 14500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:43.632434 14500 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34141
I20260928 21:14:43.632453 14628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34141 every 8 connection(s)
I20260928 21:14:43.633437 14500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:14:43.637066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14500
I20260928 21:14:43.637208 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:14:43.637502 392 external_mini_cluster.cc:1428] Setting key 9883e148786c24edaa18b3a1cb72bbed
Sep 28 21:14:43 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630083, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:43.647284 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.635306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54863 (local address 127.0.98.62:40065)
0928 21:14:43.635451 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.635455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.636341 (+ 886us) server_negotiation.cc:407] Connection header received
0928 21:14:43.637943 (+ 1602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.637947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.638009 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.638121 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:43.639376 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.639916 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.641142 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.641363 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.641476 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.643833 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:43.643837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:43.643839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:43.643872 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:43.645224 (+ 1352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.645770 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.645779 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.645780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.645853 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.646211 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.646215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.646217 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.646401 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:43.646527 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.647000 (+ 473us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.647121 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:14:43.648130 14631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.635680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:54863)
0928 21:14:43.636169 (+ 489us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.636213 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.637723 (+ 1510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.638315 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.638322 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.638710 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:43.639226 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.639239 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.640056 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.640064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.641007 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.641017 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.641513 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.641517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.641696 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.642239 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:43.642255 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:43.643668 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:43.645348 (+ 1680us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:43.645355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:43.645372 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:43.645651 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:43.645951 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:43.645954 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:43.645957 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:43.646109 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:43.646508 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:43.646515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:43.646769 (+ 254us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.646962 (+ 193us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.647309 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":109,"threads_started":1}
I20260928 21:14:43.649268 14629 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40065
I20260928 21:14:43.649487 14629 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:43.649931 14629 heartbeater.cc:511] Master 127.0.98.62:40065 requested a full tablet report, sending...
I20260928 21:14:43.651137 14170 ts_manager.cc:194] Registered new tserver with Master: 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141)
I20260928 21:14:43.651750 14170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54863
I20260928 21:14:43.663417 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:14:43.672923 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.665572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33562 (local address 127.0.98.62:40065)
0928 21:14:43.665708 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.665711 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.665921 (+ 210us) server_negotiation.cc:407] Connection header received
0928 21:14:43.666036 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.666038 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.666078 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.666147 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:43.666884 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.667392 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.668239 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.668439 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.668512 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.669898 (+ 1386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:43.669901 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:43.669903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:43.669925 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:43.671395 (+ 1470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.671782 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.671784 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.671786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.671858 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:43.672145 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:43.672147 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:43.672148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:43.672348 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:43.672447 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.672645 (+ 198us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.672740 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:14:43.675616 14170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:14:43.678717 14170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:14:43.693636 14643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.687022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37111 (local address 127.0.0.1:41488)
0928 21:14:43.688226 (+ 1204us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.688293 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.688436 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.689042 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.689048 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.689070 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:43.689225 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.689233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.690636 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.690640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.691916 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.691926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.693089 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.693092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.693211 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.693255 (+ 44us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.693366 (+ 111us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.693481 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1137,"mutex_wait_us":6,"thread_start_us":107,"threads_started":1}
I20260928 21:14:43.693846 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.687874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.0.98.1:36377)
0928 21:14:43.688299 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.688315 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.688395 (+ 80us) server_negotiation.cc:407] Connection header received
0928 21:14:43.688681 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.688688 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.688925 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.689091 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:43.689560 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.690724 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.692302 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.693234 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.693356 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.693529 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.693607 (+ 78us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.693709 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":103,"threads_started":1}
I20260928 21:14:43.694397 14641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.687286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36377 (local address 127.0.0.1:41758)
0928 21:14:43.688234 (+ 948us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.688293 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.688458 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.689266 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.689271 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.689318 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:43.689430 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.689437 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.690906 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.690911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.692175 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.692181 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.693366 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.693371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.693472 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.693496 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.693565 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.693643 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":890,"mutex_wait_us":133,"thread_start_us":89,"threads_started":1}
I20260928 21:14:43.694418 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.687750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41488 (local address 127.0.98.2:37111)
0928 21:14:43.688503 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.688517 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.688537 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:14:43.688612 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.688621 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.688876 (+ 255us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.689042 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:43.689363 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.690460 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.692053 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.692955 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.693089 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.693247 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.694148 (+ 901us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.694258 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":111,"threads_started":1}
I20260928 21:14:43.694708 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.687143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33672 (local address 127.0.98.3:34141)
0928 21:14:43.688061 (+ 918us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.688075 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.688490 (+ 415us) server_negotiation.cc:407] Connection header received
0928 21:14:43.688737 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.688742 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.688928 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.689092 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:43.689420 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.690685 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.692944 (+ 2259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.693757 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.693860 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.693985 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.694467 (+ 482us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.694568 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":154,"threads_started":1}
I20260928 21:14:43.694977 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.686978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34141 (local address 127.0.0.1:33672)
0928 21:14:43.688068 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.688411 (+ 343us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.688569 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.689211 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.689213 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.689228 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:43.689323 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.689331 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.690846 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.690853 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.692154 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.692164 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.694047 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.694052 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.694150 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.694174 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.694259 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.694333 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":7,"thread_start_us":79,"threads_started":1}
I20260928 21:14:43.697360 14563 tablet_service.cc:1467] Processing CreateTablet for tablet 59745cda566846a8b1312a436978e22c (DEFAULT_TABLE table=test-table [id=e217747e39b74fe0a3afa884cc5cce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:43.697520 14295 tablet_service.cc:1467] Processing CreateTablet for tablet 59745cda566846a8b1312a436978e22c (DEFAULT_TABLE table=test-table [id=e217747e39b74fe0a3afa884cc5cce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:43.697520 14429 tablet_service.cc:1467] Processing CreateTablet for tablet 59745cda566846a8b1312a436978e22c (DEFAULT_TABLE table=test-table [id=e217747e39b74fe0a3afa884cc5cce3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:43.698387 14563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59745cda566846a8b1312a436978e22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:43.698611 14295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59745cda566846a8b1312a436978e22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:43.698611 14429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59745cda566846a8b1312a436978e22c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:43.704723 14646 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Bootstrap starting.
I20260928 21:14:43.706382 14647 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Bootstrap starting.
I20260928 21:14:43.706822 14648 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Bootstrap starting.
I20260928 21:14:43.707351 14646 tablet_bootstrap.cc:654] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:43.708688 14646 log.cc:824] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:43.709616 14648 tablet_bootstrap.cc:654] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:43.710585 14648 log.cc:824] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:43.710817 14646 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: No bootstrap required, opened a new log
I20260928 21:14:43.711066 14646 ts_tablet_manager.cc:1397] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:14:43.711634 14647 tablet_bootstrap.cc:654] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:43.712730 14648 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: No bootstrap required, opened a new log
I20260928 21:14:43.712813 14647 log.cc:824] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:43.712955 14648 ts_tablet_manager.cc:1397] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:14:43.714416 14646 raft_consensus.cc:348] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.714666 14646 raft_consensus.cc:374] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:43.714740 14646 raft_consensus.cc:729] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4f5bdb42ea4cf8ae2ee51da3a304be, State: Initialized, Role: FOLLOWER
I20260928 21:14:43.714840 14647 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: No bootstrap required, opened a new log
I20260928 21:14:43.715024 14647 ts_tablet_manager.cc:1397] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20260928 21:14:43.715226 14646 consensus_queue.cc:260] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.715858 14629 heartbeater.cc:503] Master 127.0.98.62:40065 was elected leader, sending a full tablet report...
I20260928 21:14:43.716228 14646 ts_tablet_manager.cc:1428] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:14:43.716756 14648 raft_consensus.cc:348] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.717126 14648 raft_consensus.cc:374] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:43.717491 14648 raft_consensus.cc:729] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8825d2f0d84d919c4496ca4f013b74, State: Initialized, Role: FOLLOWER
I20260928 21:14:43.718050 14648 consensus_queue.cc:260] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.718816 14495 heartbeater.cc:503] Master 127.0.98.62:40065 was elected leader, sending a full tablet report...
I20260928 21:14:43.718860 14647 raft_consensus.cc:348] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.719087 14647 raft_consensus.cc:374] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:43.719141 14647 raft_consensus.cc:729] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d895b7a9199646e8a73a54a6559e9402, State: Initialized, Role: FOLLOWER
I20260928 21:14:43.719588 14647 consensus_queue.cc:260] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.720232 14361 heartbeater.cc:503] Master 127.0.98.62:40065 was elected leader, sending a full tablet report...
I20260928 21:14:43.720623 14647 ts_tablet_manager.cc:1428] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260928 21:14:43.721185 14648 ts_tablet_manager.cc:1428] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260928 21:14:43.809478 14653 raft_consensus.cc:482] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:43.809726 14653 raft_consensus.cc:504] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.810935 14653 leader_election.cc:288] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377)
I20260928 21:14:43.815939 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.811340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45055 (local address 127.0.98.3:34141)
0928 21:14:43.811495 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.811500 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.811531 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:14:43.811586 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.811588 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.811642 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.811727 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:43.812139 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.813015 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.814612 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.815237 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.815350 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.815412 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.815740 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.815813 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:14:43.815939 14645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.811425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40629 (local address 127.0.98.1:36377)
0928 21:14:43.811576 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.811580 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.812049 (+ 469us) server_negotiation.cc:407] Connection header received
0928 21:14:43.812146 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.812149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.812200 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.812281 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:43.812518 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.813223 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.814637 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.815237 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.815350 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.815412 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.815740 (+ 328us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.815813 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:14:43.815955 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.811220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34141 (local address 127.0.98.2:45055)
0928 21:14:43.811323 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.811340 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.811441 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.811820 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.811826 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.811851 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:43.811983 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.811994 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.813129 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.813131 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.814442 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.814453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.815368 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.815372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.815619 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.815637 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.815706 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.815808 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":187}
I20260928 21:14:43.815963 14655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.811220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36377 (local address 127.0.98.2:40629)
0928 21:14:43.811932 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.811949 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:43.812059 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:43.812297 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:43.812299 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:43.812313 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:43.812400 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.812404 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.813327 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.813330 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.814501 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:43.814512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.815368 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.815372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:43.815512 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.815614 (+ 102us) client_negotiation.cc:769] Sending connection context
0928 21:14:43.815699 (+ 85us) client_negotiation.cc:240] Negotiation successful
0928 21:14:43.815800 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":650,"mutex_wait_us":374,"thread_start_us":151,"threads_started":1}
I20260928 21:14:43.816720 14583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59745cda566846a8b1312a436978e22c" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" is_pre_election: true
I20260928 21:14:43.816812 14315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59745cda566846a8b1312a436978e22c" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d895b7a9199646e8a73a54a6559e9402" is_pre_election: true
I20260928 21:14:43.817094 14583 raft_consensus.cc:2454] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 0.
I20260928 21:14:43.817148 14315 raft_consensus.cc:2454] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 0.
I20260928 21:14:43.817582 14384 leader_election.cc:302] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be; no voters:
I20260928 21:14:43.817903 14653 raft_consensus.cc:2781] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:14:43.817981 14653 raft_consensus.cc:482] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:14:43.818011 14653 raft_consensus.cc:3037] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:43.819062 14653 raft_consensus.cc:504] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.819424 14653 leader_election.cc:288] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 election: Requested vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377)
I20260928 21:14:43.819774 14583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59745cda566846a8b1312a436978e22c" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
I20260928 21:14:43.819913 14583 raft_consensus.cc:3037] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:43.819981 14315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59745cda566846a8b1312a436978e22c" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d895b7a9199646e8a73a54a6559e9402"
I20260928 21:14:43.820133 14315 raft_consensus.cc:3037] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:43.821161 14583 raft_consensus.cc:2454] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 1.
I20260928 21:14:43.821326 14315 raft_consensus.cc:2454] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 1.
I20260928 21:14:43.821453 14384 leader_election.cc:302] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be; no voters:
I20260928 21:14:43.821635 14653 raft_consensus.cc:2781] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:43.821884 14653 raft_consensus.cc:686] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8825d2f0d84d919c4496ca4f013b74, State: Running, Role: LEADER
I20260928 21:14:43.822227 14653 consensus_queue.cc:237] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:43.825598 14169 catalog_manager.cc:5495] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0c8825d2f0d84d919c4496ca4f013b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20260928 21:14:43.846141 14644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.842741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41490 (local address 127.0.98.2:37111)
0928 21:14:43.842858 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:43.842861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:43.842874 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:14:43.842963 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:43.842966 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:43.843021 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:43.843126 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:14:43.843363 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.843900 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.844702 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:43.844980 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:43.845065 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:43.845321 (+ 256us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:14:43.845739 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:14:43.845830 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:43.845882 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:14:43.845932 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:14:43.852890 14315 raft_consensus.cc:1261] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Refusing update from remote peer 0c8825d2f0d84d919c4496ca4f013b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:43.852890 14583 raft_consensus.cc:1261] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Refusing update from remote peer 0c8825d2f0d84d919c4496ca4f013b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:43.853583 14653 consensus_queue.cc:1035] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:43.853796 14656 consensus_queue.cc:1035] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:43.862954 14659 mvcc.cc:205] Tried to move back new op lower bound from 7334420823451439104 to 7334420823338455040. Current Snapshot: MvccSnapshot[applied={T|T < 7334420823451439104 or (T in {7334420823451439104})}]
I20260928 21:14:43.863170 14660 mvcc.cc:205] Tried to move back new op lower bound from 7334420823451439104 to 7334420823338455040. Current Snapshot: MvccSnapshot[applied={T|T < 7334420823451439104}]
I20260928 21:14:43.869640 14170 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:14:43.869817 14170 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:14:43.872104 14170 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Sending DeleteTablet for 3 replicas of tablet 59745cda566846a8b1312a436978e22c
I20260928 21:14:43.872987 14429 tablet_service.cc:1514] Processing DeleteTablet for tablet 59745cda566846a8b1312a436978e22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41488
I20260928 21:14:43.872987 14295 tablet_service.cc:1514] Processing DeleteTablet for tablet 59745cda566846a8b1312a436978e22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41758
I20260928 21:14:43.873104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14232
I20260928 21:14:43.873445 14563 tablet_service.cc:1514] Processing DeleteTablet for tablet 59745cda566846a8b1312a436978e22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33672
I20260928 21:14:43.873864 14670 tablet_replica.cc:330] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: stopping tablet replica
I20260928 21:14:43.874161 14670 raft_consensus.cc:2229] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:14:43.874228 14669 tablet_replica.cc:330] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: stopping tablet replica
I20260928 21:14:43.874498 14670 raft_consensus.cc:2258] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:43.874555 14669 raft_consensus.cc:2229] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:14:43.875411 14669 raft_consensus.cc:2258] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:43.876307 14670 ts_tablet_manager.cc:1905] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:14:43.876888 14669 ts_tablet_manager.cc:1905] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:14:43.879187 14670 ts_tablet_manager.cc:1918] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:14:43.879351 14670 log.cc:1196] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/59745cda566846a8b1312a436978e22c
I20260928 21:14:43.879678 14669 ts_tablet_manager.cc:1918] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:14:43.879772 14670 ts_tablet_manager.cc:1939] T 59745cda566846a8b1312a436978e22c P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Deleting consensus metadata
I20260928 21:14:43.879871 14669 log.cc:1196] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/59745cda566846a8b1312a436978e22c
I20260928 21:14:43.880250 14669 ts_tablet_manager.cc:1939] T 59745cda566846a8b1312a436978e22c P 0c8825d2f0d84d919c4496ca4f013b74: Deleting consensus metadata
I20260928 21:14:43.881080 14156 catalog_manager.cc:4841] TS 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): tablet 59745cda566846a8b1312a436978e22c (table test-table [id=e217747e39b74fe0a3afa884cc5cce3a]) successfully deleted
I20260928 21:14:43.881311 14154 catalog_manager.cc:4841] TS 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111): tablet 59745cda566846a8b1312a436978e22c (table test-table [id=e217747e39b74fe0a3afa884cc5cce3a]) successfully deleted
W20260928 21:14:43.884879 14157 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36377 (error 108)
I20260928 21:14:43.885270 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14366
I20260928 21:14:43.886395 14640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:43.886106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36377 (local address 127.0.0.1:41766)
0928 21:14:43.886238 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:43.886322 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260928 21:14:43.886559 14157 catalog_manager.cc:4584] TS d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 59745cda566846a8b1312a436978e22c: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36377: connect: Connection refused (error 111)
I20260928 21:14:43.891791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14500
I20260928 21:14:43.897382 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14138
I20260928 21:14:43.903632 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--webserver_interface=127.0.98.62
--webserver_port=35599
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:44.013886 14672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:44.014225 14672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:44.014314 14672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:44.018404 14672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:44.018500 14672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:44.018540 14672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:44.018579 14672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:44.018618 14672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:44.023973 14672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:44.025166 14672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:44.026203 14672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:44.039592 14681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.039556 14679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.039608 14678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.039748 14672 server_base.cc:1034] running on GCE node
I20260928 21:14:44.040548 14672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:44.041822 14672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:44.043006 14672 hybrid_clock.cc:648] HybridClock initialized: now 1790630084042977 us; error 72 us; skew 500 ppm
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:44.046096 14672 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:44.047147 14672 webserver.cc:460] Webserver started at http://127.0.98.62:35599/ using document root <none> and password file <none>
I20260928 21:14:44.047814 14672 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:44.047938 14672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:44.052260 14672 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:44.054603 14688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.055915 14672 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260928 21:14:44.056085 14672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "af4c699c797649ea855b52b81c77c3b1"
format_stamp: "Formatted at 2026-09-28 21:14:42 on dist-test-slave-v7xz"
server_key: "207b881ba976b69e12f937e834f1edf1"
server_key_iv: "ff434ac2a4567d21c58485547d5d3797"
server_key_version: "encryptionkey@0"
I20260928 21:14:44.056737 14672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:44.076059 14672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:44.079605 14672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:44.079821 14672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:44.087551 14672 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40065
I20260928 21:14:44.087874 14740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40065 every 8 connection(s)
I20260928 21:14:44.088735 14672 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:44.090263 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14672
I20260928 21:14:44.090775 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36377
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40065
--webserver_port=37321
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:14:44.094131 14741 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:14:44.095846 14741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap starting.
I20260928 21:14:44.112994 14741 log.cc:824] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:44.121788 14741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:14:44.122237 14741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap complete.
I20260928 21:14:44.126381 14741 raft_consensus.cc:348] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:44.126778 14741 raft_consensus.cc:729] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af4c699c797649ea855b52b81c77c3b1, State: Initialized, Role: FOLLOWER
I20260928 21:14:44.127298 14741 consensus_queue.cc:260] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:44.127416 14741 raft_consensus.cc:388] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:14:44.127532 14741 raft_consensus.cc:482] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:14:44.127646 14741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:14:44.129666 14741 raft_consensus.cc:504] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:44.130026 14741 leader_election.cc:302] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af4c699c797649ea855b52b81c77c3b1; no voters:
I20260928 21:14:44.130365 14741 leader_election.cc:288] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:14:44.130488 14745 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:14:44.130740 14745 raft_consensus.cc:686] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 LEADER]: Becoming Leader. State: Replica: af4c699c797649ea855b52b81c77c3b1, State: Running, Role: LEADER
I20260928 21:14:44.131117 14745 consensus_queue.cc:237] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:44.131637 14741 sys_catalog.cc:564] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:44.132642 14747 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af4c699c797649ea855b52b81c77c3b1. Latest consensus state: current_term: 2 leader_uuid: "af4c699c797649ea855b52b81c77c3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } }
I20260928 21:14:44.132691 14746 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "af4c699c797649ea855b52b81c77c3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } }
I20260928 21:14:44.132792 14746 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:44.132778 14747 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:44.136686 14760 catalog_manager.cc:1259] Loaded cluster ID: 6e511739f3de4faaa1c28b13217d4dfa
I20260928 21:14:44.136768 14760 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: loading cluster ID for follower catalog manager: success
W20260928 21:14:44.137617 14760 catalog_manager.cc:1576] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260928 21:14:44.137702 14760 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260928 21:14:44.137758 14754 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:44.139698 14754 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:44.140139 14754 catalog_manager.cc:1259] Loaded cluster ID: 6e511739f3de4faaa1c28b13217d4dfa
I20260928 21:14:44.140216 14754 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:14:44.141144 14754 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260928 21:14:44.143865 14754 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260928 21:14:44.148301 14754 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:14:44.149084 14754 catalog_manager.cc:5879] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Loaded TSK: 0
I20260928 21:14:44.150055 14754 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:14:44.218896 14743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:44.219206 14743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:44.219282 14743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:44.223397 14743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:44.223487 14743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:44.223585 14743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:14:44.228994 14743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37321
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:44.230020 14743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:44.231070 14743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:44.242239 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.242328 14743 server_base.cc:1034] running on GCE node
W20260928 21:14:44.242242 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.242239 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.242951 14743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:44.243660 14743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:44.244892 14743 hybrid_clock.cc:648] HybridClock initialized: now 1790630084244853 us; error 46 us; skew 500 ppm
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:44.247830 14743 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:14:44.248805 14743 webserver.cc:460] Webserver started at http://127.0.98.1:37321/ using document root <none> and password file <none>
I20260928 21:14:44.249385 14743 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:44.249488 14743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:44.253351 14743 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:14:44.255568 14778 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.257089 14743 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:14:44.257233 14743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "d895b7a9199646e8a73a54a6559e9402"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "bbbd5d55adb9a6f3153b11a1e37be3a3"
server_key_iv: "db6e8f68ca2645caf8b127b607067901"
server_key_version: "encryptionkey@0"
I20260928 21:14:44.257838 14743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:44.279994 14743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:44.283346 14743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:44.283557 14743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:44.284207 14743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:44.285432 14785 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:14:44.289606 14743 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:14:44.289690 14743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 21:14:44.289795 14743 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:14:44.291456 14743 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:14:44.291527 14743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:14:44.291790 14785 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Bootstrap starting.
I20260928 21:14:44.302527 14743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36377
I20260928 21:14:44.302582 14892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36377 every 8 connection(s)
I20260928 21:14:44.303472 14743 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:14:44.307976 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14743
I20260928 21:14:44.308527 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37111
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40065
--webserver_port=36791
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:14:44.310843 14785 log.cc:824] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:44.320705 14785 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:14:44.320936 14896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.305454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:33599)
0928 21:14:44.306077 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.306152 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.306964 (+ 812us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.308387 (+ 1423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.308396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.308941 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:44.309672 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.309696 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.311629 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.311633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.312537 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.312548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.313404 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.313408 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.313602 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.314222 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:44.314238 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:44.315701 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:44.317844 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.317849 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.317862 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.318128 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.318478 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.318480 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.318482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.318617 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.319179 (+ 562us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:44.319185 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:44.319436 (+ 251us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.319620 (+ 184us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.319954 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":150,"threads_started":1}
I20260928 21:14:44.320956 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.305541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33599 (local address 127.0.98.62:40065)
0928 21:14:44.305981 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.306004 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.306287 (+ 283us) server_negotiation.cc:407] Connection header received
0928 21:14:44.307457 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.307474 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.307905 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.308314 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:44.309887 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.311424 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.312668 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.313249 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.313429 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.315896 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:44.315901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:44.315913 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:44.315942 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:44.317716 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.318252 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.318258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.318262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.318374 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.318712 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.318716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.318718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.319054 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:44.319194 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.319730 (+ 536us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.320006 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":124,"threads_started":1}
I20260928 21:14:44.321206 14785 tablet_bootstrap.cc:492] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Bootstrap complete.
I20260928 21:14:44.322563 14785 ts_tablet_manager.cc:1397] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Time spent bootstrapping tablet: real 0.031s user 0.029s sys 0.000s
I20260928 21:14:44.323756 14893 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40065
I20260928 21:14:44.323992 14893 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:44.324512 14893 heartbeater.cc:511] Master 127.0.98.62:40065 requested a full tablet report, sending...
I20260928 21:14:44.325263 14785 raft_consensus.cc:348] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.325681 14785 raft_consensus.cc:729] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d895b7a9199646e8a73a54a6559e9402, State: Initialized, Role: FOLLOWER
I20260928 21:14:44.326169 14785 consensus_queue.cc:260] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.327004 14785 ts_tablet_manager.cc:1428] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 21:14:44.327186 14705 ts_manager.cc:194] Registered new tserver with Master: d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377)
I20260928 21:14:44.331198 14705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33599
I20260928 21:14:44.334162 14893 heartbeater.cc:503] Master 127.0.98.62:40065 was elected leader, sending a full tablet report...
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:14:44.342108 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.330482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36377 (local address 127.0.0.1:41780)
0928 21:14:44.331192 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.331241 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.331472 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.332023 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.332029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.332490 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:44.332622 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.332631 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.334765 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.334772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.335724 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.335734 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.336300 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.336317 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.336442 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.337046 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:44.337059 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:44.338376 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:44.340317 (+ 1941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.340326 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.340332 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.340567 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.340907 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.340910 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.340912 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.340999 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.341452 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:44.341458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:44.341585 (+ 127us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.341721 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.341917 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":123,"threads_started":1}
I20260928 21:14:44.342113 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.330519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41780 (local address 127.0.98.1:36377)
0928 21:14:44.330920 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.330930 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.331365 (+ 435us) server_negotiation.cc:407] Connection header received
0928 21:14:44.331667 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.331672 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.331874 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.332023 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:44.332769 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.334649 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.335840 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.336188 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.336311 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.338518 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:44.338528 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:44.338550 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:44.338571 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:44.340199 (+ 1628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.340684 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.340690 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.340692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.340776 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.341098 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.341102 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.341104 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.341334 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:44.341478 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.341813 (+ 335us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.341968 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":313,"thread_start_us":103,"threads_started":1}
I20260928 21:14:44.343137 14827 tablet_service.cc:1514] Processing DeleteTablet for tablet 59745cda566846a8b1312a436978e22c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41780
I20260928 21:14:44.343971 14902 tablet_replica.cc:330] stopping tablet replica
I20260928 21:14:44.344300 14902 raft_consensus.cc:2229] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:14:44.344549 14902 raft_consensus.cc:2258] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:44.345636 14902 ts_tablet_manager.cc:1905] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:14:44.350291 14902 ts_tablet_manager.cc:1918] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:14:44.350400 14902 log.cc:1196] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/59745cda566846a8b1312a436978e22c
I20260928 21:14:44.350666 14902 ts_tablet_manager.cc:1939] T 59745cda566846a8b1312a436978e22c P d895b7a9199646e8a73a54a6559e9402: Deleting consensus metadata
I20260928 21:14:44.352629 14692 catalog_manager.cc:4841] TS d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377): tablet 59745cda566846a8b1312a436978e22c (table test-table [id=e217747e39b74fe0a3afa884cc5cce3a]) successfully deleted
W20260928 21:14:44.431133 14897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:44.431380 14897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:44.431421 14897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:44.437948 14897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:44.438011 14897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:44.438105 14897 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:14:44.443450 14897 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36791
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:44.444504 14897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:44.445626 14897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:44.455032 14908 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.455016 14907 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.455089 14897 server_base.cc:1034] running on GCE node
W20260928 21:14:44.455009 14910 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.455657 14897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:44.456307 14897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:44.457474 14897 hybrid_clock.cc:648] HybridClock initialized: now 1790630084457455 us; error 29 us; skew 500 ppm
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:44.460108 14897 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:14:44.461064 14897 webserver.cc:460] Webserver started at http://127.0.98.2:36791/ using document root <none> and password file <none>
I20260928 21:14:44.461673 14897 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:44.461743 14897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:44.465799 14897 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:14:44.467860 14917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.469262 14897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:14:44.469395 14897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "0c8825d2f0d84d919c4496ca4f013b74"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "3384c29610fe84e7feaad23d81a5aa46"
server_key_iv: "c0d3ef3cacca8bf52fa98a6b45f49e11"
server_key_version: "encryptionkey@0"
I20260928 21:14:44.469983 14897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:44.486177 14897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:44.489377 14897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:44.489602 14897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:44.490247 14897 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:44.491175 14897 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:44.491225 14897 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.491291 14897 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:44.491326 14897 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.500715 15030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37111 every 8 connection(s)
I20260928 21:14:44.500720 14897 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37111
I20260928 21:14:44.501870 14897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:14:44.505796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 14897
I20260928 21:14:44.506266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34141
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40065
--webserver_port=36865
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:44.517654 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.503753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35655 (local address 127.0.98.62:40065)
0928 21:14:44.503926 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.503931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.504788 (+ 857us) server_negotiation.cc:407] Connection header received
0928 21:14:44.506173 (+ 1385us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.506179 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.506248 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.506351 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:44.507792 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.508328 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.509678 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.510110 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.510272 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.513182 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:44.513198 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:44.513203 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:44.513243 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:44.515062 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.515615 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.515622 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.515626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.515727 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.516161 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.516167 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.516171 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.516462 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:44.516581 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.517235 (+ 654us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.517403 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:14:44.518379 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.504170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:35655)
0928 21:14:44.504628 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.504672 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.505957 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.506594 (+ 637us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.506601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.506966 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:44.507612 (+ 646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.507636 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.508464 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.508470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.509541 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.509552 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.510301 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.510304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.510455 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.511126 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:44.511143 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:44.512992 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:44.515189 (+ 2197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.515194 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.515208 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.515491 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.515866 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.515869 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.515870 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.516043 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.516583 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:44.516589 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:44.516932 (+ 343us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.517122 (+ 190us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.517411 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260928 21:14:44.519611 15031 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40065
I20260928 21:14:44.519884 15031 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:44.520460 15031 heartbeater.cc:511] Master 127.0.98.62:40065 requested a full tablet report, sending...
I20260928 21:14:44.521765 14705 ts_manager.cc:194] Registered new tserver with Master: 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
I20260928 21:14:44.522537 14705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35655
W20260928 21:14:44.622988 15034 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:44.623265 15034 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:44.623309 15034 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:44.627332 15034 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:44.627414 15034 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:44.627476 15034 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:14:44.633093 15034 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36865
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15034
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:44.634367 15034 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:44.635356 15034 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:44.644902 15042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.644941 15040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.644901 15039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.645236 15034 server_base.cc:1034] running on GCE node
I20260928 21:14:44.645627 15034 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:44.646292 15034 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:44.647478 15034 hybrid_clock.cc:648] HybridClock initialized: now 1790630084647458 us; error 35 us; skew 500 ppm
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:44.650391 15034 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:14:44.651355 15034 webserver.cc:460] Webserver started at http://127.0.98.3:36865/ using document root <none> and password file <none>
I20260928 21:14:44.651932 15034 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:44.652029 15034 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:44.656023 15034 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:44.658259 15049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.659554 15034 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:14:44.659708 15034 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "b2a9cb6252460ec78032998be15891c7"
server_key_iv: "c173984acd3707d400c6be6c12833513"
server_key_version: "encryptionkey@0"
I20260928 21:14:44.660332 15034 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:44.676159 15034 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:44.679436 15034 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:44.679656 15034 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:44.680258 15034 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:44.681130 15034 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:44.681198 15034 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.681259 15034 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:44.681311 15034 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.690377 15034 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34141
I20260928 21:14:44.690394 15162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34141 every 8 connection(s)
I20260928 21:14:44.691352 15034 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:14:44.693032 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15034
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:14:44.702831 15165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.693685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33572 (local address 127.0.98.62:40065)
0928 21:14:44.693923 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.693927 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.694128 (+ 201us) server_negotiation.cc:407] Connection header received
0928 21:14:44.694289 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.694292 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.694344 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.694446 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:44.695462 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.696024 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.696980 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.697210 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.697328 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.699016 (+ 1688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:44.699019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:44.699021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:44.699044 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:44.700812 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.701319 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.701324 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.701326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.701425 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.701747 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.701751 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.701754 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.701969 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:44.702201 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.702506 (+ 305us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.702632 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":72,"threads_started":1}
I20260928 21:14:44.706383 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.693143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49447 (local address 127.0.98.62:40065)
0928 21:14:44.693288 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.693293 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.694237 (+ 944us) server_negotiation.cc:407] Connection header received
0928 21:14:44.695879 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.695883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.695935 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.696031 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:44.697257 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.697851 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.699424 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.699708 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.699787 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.702606 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:44.702611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:44.702619 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:44.702640 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:44.704472 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.705101 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.705103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.705105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.705178 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:44.705481 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:44.705483 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:44.705484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:44.705642 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:44.705713 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.706165 (+ 452us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.706271 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:14:44.707237 15167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.693577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:49447)
0928 21:14:44.694085 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.694140 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.695666 (+ 1526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.696222 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.696229 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.696571 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:44.697123 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.697137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.697993 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.697996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.699313 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.699323 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.699873 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.699878 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.700081 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.700593 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:44.700618 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:44.702430 (+ 1812us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:44.704611 (+ 2181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.704618 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.704637 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.705014 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.705264 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:44.705266 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:44.705267 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:44.705398 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:44.705728 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:44.705733 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:44.705960 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.706105 (+ 145us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.706370 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":132,"threads_started":1}
I20260928 21:14:44.708328 15163 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40065
I20260928 21:14:44.708577 15163 heartbeater.cc:464] Registering TS with master...
I20260928 21:14:44.709138 15163 heartbeater.cc:511] Master 127.0.98.62:40065 requested a full tablet report, sending...
I20260928 21:14:44.710125 14705 ts_manager.cc:194] Registered new tserver with Master: 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141)
I20260928 21:14:44.710683 14705 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49447
I20260928 21:14:44.716506 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:14:44.721171 14895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.717209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.98.62:40065)
0928 21:14:44.717364 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.717369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.717571 (+ 202us) server_negotiation.cc:407] Connection header received
0928 21:14:44.717768 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.717771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.717885 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.717995 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:14:44.718260 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.718779 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.719653 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.719896 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.720010 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.720331 (+ 321us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:14:44.720734 (+ 403us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:14:44.720912 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.720969 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.721022 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:14:44.721843 14705 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:14:44.725206 14705 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:14:44.733198 14827 tablet_service.cc:1467] Processing CreateTablet for tablet 039f7da368584c62a26ea9847a99291d (DEFAULT_TABLE table=test-table [id=1ebe083044bf41618b3bad27aa59adf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:44.733930 14827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039f7da368584c62a26ea9847a99291d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:44.736806 14785 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Bootstrap starting.
I20260928 21:14:44.737742 14785 tablet_bootstrap.cc:654] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:44.737914 14900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.731999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34141 (local address 127.0.0.1:33678)
0928 21:14:44.732122 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.732140 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.732217 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.733277 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.733280 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.733300 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:44.733411 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.733417 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.734991 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.734994 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.736270 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.736281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.737464 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.737468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.737561 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.737602 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.737691 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.737777 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":33}
I20260928 21:14:44.737952 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.732241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33678 (local address 127.0.98.3:34141)
0928 21:14:44.732683 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.732696 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.732721 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:14:44.732853 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.732859 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.733093 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.733603 (+ 510us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:44.733773 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.734839 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.736413 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.737332 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.737464 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.737617 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.737722 (+ 105us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.737821 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":187,"threads_started":1}
I20260928 21:14:44.738399 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.732784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37111 (local address 127.0.0.1:41496)
0928 21:14:44.733135 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.733148 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.733241 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.734104 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.734108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.734128 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:44.734212 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.734217 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.735657 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.735660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.736711 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.736722 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.737960 (+ 1238us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.737964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.738081 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.738113 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.738173 (+ 60us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.738257 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":1,"thread_start_us":127,"threads_started":1}
I20260928 21:14:44.739028 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.733129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41496 (local address 127.0.98.2:37111)
0928 21:14:44.733565 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.733587 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.733610 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:14:44.733714 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.733720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.733925 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.734329 (+ 404us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:44.734450 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.735500 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.737004 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.737813 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.738568 (+ 755us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.738758 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.738819 (+ 61us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.738899 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":157,"threads_started":1}
I20260928 21:14:44.740213 14785 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: No bootstrap required, opened a new log
I20260928 21:14:44.740331 14785 ts_tablet_manager.cc:1397] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.002s
I20260928 21:14:44.740756 14785 raft_consensus.cc:348] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.740931 14785 raft_consensus.cc:374] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:44.740980 14785 raft_consensus.cc:729] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d895b7a9199646e8a73a54a6559e9402, State: Initialized, Role: FOLLOWER
I20260928 21:14:44.741104 14785 consensus_queue.cc:260] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.741385 14785 ts_tablet_manager.cc:1428] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260928 21:14:44.741300 15097 tablet_service.cc:1467] Processing CreateTablet for tablet 039f7da368584c62a26ea9847a99291d (DEFAULT_TABLE table=test-table [id=1ebe083044bf41618b3bad27aa59adf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:44.741927 14965 tablet_service.cc:1467] Processing CreateTablet for tablet 039f7da368584c62a26ea9847a99291d (DEFAULT_TABLE table=test-table [id=1ebe083044bf41618b3bad27aa59adf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:14:44.742458 15097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039f7da368584c62a26ea9847a99291d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:44.742964 14965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 039f7da368584c62a26ea9847a99291d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:14:44.750839 15176 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Bootstrap starting.
I20260928 21:14:44.751623 15177 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Bootstrap starting.
I20260928 21:14:44.754158 15176 tablet_bootstrap.cc:654] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:44.754158 15177 tablet_bootstrap.cc:654] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Neither blocks nor log segments found. Creating new log.
I20260928 21:14:44.755129 15177 log.cc:824] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:44.755129 15176 log.cc:824] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:44.757287 15177 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: No bootstrap required, opened a new log
I20260928 21:14:44.757601 15177 ts_tablet_manager.cc:1397] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:14:44.758224 15176 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: No bootstrap required, opened a new log
I20260928 21:14:44.758407 15176 ts_tablet_manager.cc:1397] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260928 21:14:44.761715 15177 raft_consensus.cc:348] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.762056 15177 raft_consensus.cc:374] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:44.762128 15177 raft_consensus.cc:729] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8825d2f0d84d919c4496ca4f013b74, State: Initialized, Role: FOLLOWER
I20260928 21:14:44.762207 15176 raft_consensus.cc:348] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.762498 15176 raft_consensus.cc:374] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:14:44.762558 15176 raft_consensus.cc:729] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f4f5bdb42ea4cf8ae2ee51da3a304be, State: Initialized, Role: FOLLOWER
I20260928 21:14:44.762632 15177 consensus_queue.cc:260] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.762971 15176 consensus_queue.cc:260] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.763440 15031 heartbeater.cc:503] Master 127.0.98.62:40065 was elected leader, sending a full tablet report...
I20260928 21:14:44.763619 15163 heartbeater.cc:503] Master 127.0.98.62:40065 was elected leader, sending a full tablet report...
I20260928 21:14:44.763803 15177 ts_tablet_manager.cc:1428] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 21:14:44.763818 15176 ts_tablet_manager.cc:1428] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:14:44.765769 15180 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:44.766027 15180 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.767321 15180 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377)
I20260928 21:14:44.772184 15172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.767846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42601 (local address 127.0.98.3:34141)
0928 21:14:44.767949 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.767954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.767969 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:14:44.768024 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.768027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.768080 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.768154 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:44.768542 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.769284 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.770832 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.771595 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.771726 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.771790 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.771989 (+ 199us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.772061 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260928 21:14:44.772184 15033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.767520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34141 (local address 127.0.98.2:42601)
0928 21:14:44.767622 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.767652 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.767734 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.768274 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.768277 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.768289 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:44.768417 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.768424 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.769390 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.769392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.770691 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.770702 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.771739 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.771743 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.771857 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.771885 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.771975 (+ 90us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.772051 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":26}
I20260928 21:14:44.772249 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.767910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51467 (local address 127.0.98.1:36377)
0928 21:14:44.768049 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.768056 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.768124 (+ 68us) server_negotiation.cc:407] Connection header received
0928 21:14:44.768249 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.768253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.768318 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.768413 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:14:44.768801 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.769560 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.770948 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.771687 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.771819 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.772010 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.772049 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.772120 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:14:44.772305 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.767537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36377 (local address 127.0.98.2:51467)
0928 21:14:44.768051 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.768063 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:44.768157 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:44.768522 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:44.768525 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:44.768542 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:14:44.768674 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.768681 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.769667 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.769671 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.770826 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:44.770833 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.771820 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.771823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:44.771928 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.771949 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:14:44.772012 (+ 63us) client_negotiation.cc:240] Negotiation successful
0928 21:14:44.772112 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":230,"thread_start_us":109,"threads_started":1}
I20260928 21:14:44.773090 15117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039f7da368584c62a26ea9847a99291d" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" is_pre_election: true
I20260928 21:14:44.773231 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039f7da368584c62a26ea9847a99291d" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d895b7a9199646e8a73a54a6559e9402" is_pre_election: true
I20260928 21:14:44.773423 15117 raft_consensus.cc:2454] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 0.
I20260928 21:14:44.773584 14847 raft_consensus.cc:2454] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 0.
I20260928 21:14:44.773948 14920 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be; no voters:
I20260928 21:14:44.774300 15180 raft_consensus.cc:2781] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:14:44.774384 15180 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:14:44.774432 15180 raft_consensus.cc:3037] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:44.775887 15180 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.776238 15180 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 election: Requested vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377)
I20260928 21:14:44.776571 15117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039f7da368584c62a26ea9847a99291d" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
I20260928 21:14:44.776580 14847 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "039f7da368584c62a26ea9847a99291d" candidate_uuid: "0c8825d2f0d84d919c4496ca4f013b74" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d895b7a9199646e8a73a54a6559e9402"
I20260928 21:14:44.776710 15117 raft_consensus.cc:3037] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:44.776724 14847 raft_consensus.cc:3037] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:14:44.777787 15117 raft_consensus.cc:2454] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 1.
I20260928 21:14:44.777794 14847 raft_consensus.cc:2454] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0c8825d2f0d84d919c4496ca4f013b74 in term 1.
I20260928 21:14:44.778081 14920 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be; no voters:
I20260928 21:14:44.778297 15180 raft_consensus.cc:2781] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:14:44.778504 15180 raft_consensus.cc:686] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8825d2f0d84d919c4496ca4f013b74, State: Running, Role: LEADER
I20260928 21:14:44.778875 15180 consensus_queue.cc:237] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:44.781970 14705 catalog_manager.cc:5495] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "0c8825d2f0d84d919c4496ca4f013b74" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } health_report { overall_health: UNKNOWN } } }
I20260928 21:14:44.794023 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.790193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41508 (local address 127.0.98.2:37111)
0928 21:14:44.790322 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:44.790325 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:44.790340 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:14:44.790491 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:44.790493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:44.790550 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:44.790641 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:14:44.790949 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.791447 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.792310 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:44.792557 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:44.792642 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:44.793108 (+ 466us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:14:44.793526 (+ 418us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:14:44.793630 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:44.793766 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 21:14:44.793818 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:14:44.799841 15117 raft_consensus.cc:1261] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Refusing update from remote peer 0c8825d2f0d84d919c4496ca4f013b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:44.799837 14847 raft_consensus.cc:1261] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Refusing update from remote peer 0c8825d2f0d84d919c4496ca4f013b74: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:14:44.800575 15180 consensus_queue.cc:1035] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:14:44.800745 15183 consensus_queue.cc:1035] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:14:44.804651 14894 tablet.cc:2369] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:14:44.806309 15186 mvcc.cc:205] Tried to move back new op lower bound from 7334420827332866048 to 7334420827257016320. Current Snapshot: MvccSnapshot[applied={T|T < 7334420827332866048}]
I20260928 21:14:44.806409 15185 mvcc.cc:205] Tried to move back new op lower bound from 7334420827332866048 to 7334420827257016320. Current Snapshot: MvccSnapshot[applied={T|T < 7334420827332866048}]
I20260928 21:14:44.811691 15187 mvcc.cc:205] Tried to move back new op lower bound from 7334420827332866048 to 7334420827257016320. Current Snapshot: MvccSnapshot[applied={T|T < 7334420827332866048}]
I20260928 21:14:44.815354 14704 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:14:44.815516 14704 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:33580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:14:44.817888 14704 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Sending DeleteTablet for 3 replicas of tablet 039f7da368584c62a26ea9847a99291d
I20260928 21:14:44.818528 15097 tablet_service.cc:1514] Processing DeleteTablet for tablet 039f7da368584c62a26ea9847a99291d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33678
I20260928 21:14:44.818733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14743
I20260928 21:14:44.818712 14965 tablet_service.cc:1514] Processing DeleteTablet for tablet 039f7da368584c62a26ea9847a99291d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:14:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41496
I20260928 21:14:44.821079 15196 tablet_replica.cc:330] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: stopping tablet replica
I20260928 21:14:44.821424 15196 raft_consensus.cc:2229] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:14:44.821779 15196 raft_consensus.cc:2258] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:44.822155 15195 tablet_replica.cc:330] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: stopping tablet replica
I20260928 21:14:44.822485 15195 raft_consensus.cc:2229] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:14:44.822796 15195 raft_consensus.cc:2258] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:14:44.823400 15196 ts_tablet_manager.cc:1905] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:14:44.824328 15195 ts_tablet_manager.cc:1905] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260928 21:14:44.825591 14692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36377 (error 108)
I20260928 21:14:44.826232 15196 ts_tablet_manager.cc:1918] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:14:44.826336 15196 log.cc:1196] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/039f7da368584c62a26ea9847a99291d
I20260928 21:14:44.826730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14897
I20260928 21:14:44.826790 15196 ts_tablet_manager.cc:1939] T 039f7da368584c62a26ea9847a99291d P 0c8825d2f0d84d919c4496ca4f013b74: Deleting consensus metadata
I20260928 21:14:44.827637 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.827037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36377 (local address 127.0.0.1:41792)
0928 21:14:44.827506 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.827567 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361}
W20260928 21:14:44.828311 14692 catalog_manager.cc:4584] TS d895b7a9199646e8a73a54a6559e9402 (127.0.98.1:36377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 039f7da368584c62a26ea9847a99291d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36377: connect: Connection refused (error 111)
I20260928 21:14:44.829396 15195 ts_tablet_manager.cc:1918] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:14:44.829517 15195 log.cc:1196] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/039f7da368584c62a26ea9847a99291d
I20260928 21:14:44.829870 15195 ts_tablet_manager.cc:1939] T 039f7da368584c62a26ea9847a99291d P 9f4f5bdb42ea4cf8ae2ee51da3a304be: Deleting consensus metadata
I20260928 21:14:44.831219 14691 catalog_manager.cc:4841] TS 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): tablet 039f7da368584c62a26ea9847a99291d (table test-table [id=1ebe083044bf41618b3bad27aa59adf9]) successfully deleted
I20260928 21:14:44.834234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15034
I20260928 21:14:44.834473 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:44.834162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37111 (local address 127.0.0.1:41522)
0928 21:14:44.834310 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:44.834379 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:37111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 21:14:44.839869 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 14672
I20260928 21:14:44.846287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--webserver_interface=127.0.98.62
--webserver_port=35599
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40065
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:44.959087 15198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:44.959439 15198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:44.959522 15198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:44.963615 15198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:14:44.963716 15198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:44.963762 15198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:44.963814 15198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:14:44.963852 15198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:14:44.969355 15198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40065
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=35599
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:44.970464 15198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:44.971550 15198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:44.980376 15203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.980373 15204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:44.980373 15206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:44.980757 15198 server_base.cc:1034] running on GCE node
I20260928 21:14:44.981365 15198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:44.982491 15198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:44.983680 15198 hybrid_clock.cc:648] HybridClock initialized: now 1790630084983663 us; error 32 us; skew 500 ppm
Sep 28 21:14:44 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:44.986483 15198 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:14:44.987401 15198 webserver.cc:460] Webserver started at http://127.0.98.62:35599/ using document root <none> and password file <none>
I20260928 21:14:44.987968 15198 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:44.988062 15198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:44.991917 15198 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:14:44.994102 15213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:44.995374 15198 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:14:44.995609 15198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "af4c699c797649ea855b52b81c77c3b1"
format_stamp: "Formatted at 2026-09-28 21:14:42 on dist-test-slave-v7xz"
server_key: "207b881ba976b69e12f937e834f1edf1"
server_key_iv: "ff434ac2a4567d21c58485547d5d3797"
server_key_version: "encryptionkey@0"
I20260928 21:14:44.996176 15198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:45.008962 15198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:45.011790 15198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:45.011984 15198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:45.019234 15198 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40065
I20260928 21:14:45.019543 15265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40065 every 8 connection(s)
I20260928 21:14:45.020285 15198 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:14:45.022567 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15198
I20260928 21:14:45.023093 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:36377
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40065
--webserver_port=37321
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:14:45.025607 15266 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:14:45.027418 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap starting.
I20260928 21:14:45.046262 15266 log.cc:824] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:45.058804 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:14:45.059262 15266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1: Bootstrap complete.
I20260928 21:14:45.063890 15266 raft_consensus.cc:348] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:45.064421 15266 raft_consensus.cc:729] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af4c699c797649ea855b52b81c77c3b1, State: Initialized, Role: FOLLOWER
I20260928 21:14:45.065049 15266 consensus_queue.cc:260] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:45.065225 15266 raft_consensus.cc:388] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:14:45.065318 15266 raft_consensus.cc:482] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:14:45.065447 15266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:14:45.067770 15266 raft_consensus.cc:504] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:45.068140 15266 leader_election.cc:302] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af4c699c797649ea855b52b81c77c3b1; no voters:
I20260928 21:14:45.068507 15266 leader_election.cc:288] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:14:45.068598 15270 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:14:45.068850 15270 raft_consensus.cc:686] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [term 3 LEADER]: Becoming Leader. State: Replica: af4c699c797649ea855b52b81c77c3b1, State: Running, Role: LEADER
I20260928 21:14:45.069190 15270 consensus_queue.cc:237] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } }
I20260928 21:14:45.069908 15266 sys_catalog.cc:564] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:14:45.070219 15270 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af4c699c797649ea855b52b81c77c3b1. Latest consensus state: current_term: 3 leader_uuid: "af4c699c797649ea855b52b81c77c3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } }
I20260928 21:14:45.070354 15270 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:45.070742 15271 sys_catalog.cc:455] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "af4c699c797649ea855b52b81c77c3b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af4c699c797649ea855b52b81c77c3b1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40065 } } }
I20260928 21:14:45.070923 15271 sys_catalog.cc:458] T 00000000000000000000000000000000 P af4c699c797649ea855b52b81c77c3b1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:14:45.072033 15274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:14:45.077342 15274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:14:45.077988 15274 catalog_manager.cc:1259] Loaded cluster ID: 6e511739f3de4faaa1c28b13217d4dfa
I20260928 21:14:45.078050 15274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260928 21:14:45.079099 15274 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe6e0f529ad google::LogMessage::Fail() at ??:0
@ 0x7fe6e0f5588a google::LogMessage::SendToLog() at ??:0
@ 0x7fe6e0f524de google::LogMessage::Flush() at ??:0
@ 0x7fe6e0f53a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe6eb891fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7fe6eb892d2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe6eb88f579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7fe6eb8d3640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe6e31a43d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe6e31a4c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7fe6e31a63f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7fe6e3195512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe6e47d96db start_thread at ??:0
@ 0x7fe6dfa5061f clone at ??:0
W20260928 21:14:45.180471 15268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:45.180953 15268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:45.181042 15268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:45.188107 15268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:45.188211 15268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:45.188318 15268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:14:45.194445 15268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:36377
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=37321
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:45.195674 15268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:45.197103 15268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:45.207645 15308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:45.207669 15307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:45.207649 15310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:45.208169 15268 server_base.cc:1034] running on GCE node
I20260928 21:14:45.208575 15268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:45.209290 15268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:45.210477 15268 hybrid_clock.cc:648] HybridClock initialized: now 1790630085210440 us; error 49 us; skew 500 ppm
Sep 28 21:14:45 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:45.213507 15268 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:14:45.214607 15268 webserver.cc:460] Webserver started at http://127.0.98.1:37321/ using document root <none> and password file <none>
I20260928 21:14:45.215201 15268 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:45.215297 15268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:45.219301 15268 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:14:45.221686 15317 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.223018 15268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:14:45.223155 15268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "d895b7a9199646e8a73a54a6559e9402"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "bbbd5d55adb9a6f3153b11a1e37be3a3"
server_key_iv: "db6e8f68ca2645caf8b127b607067901"
server_key_version: "encryptionkey@0"
I20260928 21:14:45.223786 15268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:45.237411 15268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:45.240689 15268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:45.240940 15268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:45.241547 15268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:45.242647 15324 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:14:45.248137 15268 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:14:45.248241 15268 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260928 21:14:45.248349 15268 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:14:45.250028 15268 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:14:45.250078 15268 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.001s
I20260928 21:14:45.250495 15324 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Bootstrap starting.
I20260928 21:14:45.261420 15268 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36377
I20260928 21:14:45.261467 15431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36377 every 8 connection(s)
I20260928 21:14:45.262470 15268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:14:45.270995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15268
I20260928 21:14:45.271451 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:37111
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40065
--webserver_port=36791
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:14:45.273293 15324 log.cc:824] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Log is configured to *not* fsync() on all Append() calls
I20260928 21:14:45.282091 15324 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:14:45.282526 15324 tablet_bootstrap.cc:492] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Bootstrap complete.
I20260928 21:14:45.283839 15324 ts_tablet_manager.cc:1397] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Time spent bootstrapping tablet: real 0.034s user 0.019s sys 0.009s
I20260928 21:14:45.287147 15324 raft_consensus.cc:348] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:45.287712 15324 raft_consensus.cc:729] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d895b7a9199646e8a73a54a6559e9402, State: Initialized, Role: FOLLOWER
I20260928 21:14:45.288271 15324 consensus_queue.cc:260] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:45.289423 15324 ts_tablet_manager.cc:1428] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
W20260928 21:14:45.336989 15434 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:14:45.264694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:59569)
0928 21:14:45.267346 (+ 2652us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.267391 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:45.268354 (+ 963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:45.336119 (+ 67765us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":2453,"thread_start_us":200,"threads_started":1}
W20260928 21:14:45.337502 15432 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40065 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40065: Client connection negotiation failed: client connection to 127.0.98.62:40065: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:14:45.338303 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.338035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:38829)
0928 21:14:45.338168 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.338234 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:14:45.338960 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.338732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:38909)
0928 21:14:45.338857 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.338911 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:14:45.339151 15432 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:14:45.387243 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:45.387562 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:45.387672 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:45.391702 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:45.391793 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:45.391889 15435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:14:45.397187 15435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:37111
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36791
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:45.398267 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:45.399287 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:14:45.409596 15435 server_base.cc:1034] running on GCE node
W20260928 21:14:45.409636 15442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:45.409668 15445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:45.409790 15443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:45.410205 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:45.410796 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:45.412006 15435 hybrid_clock.cc:648] HybridClock initialized: now 1790630085411990 us; error 30 us; skew 500 ppm
Sep 28 21:14:45 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:45.415148 15435 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:14:45.416103 15435 webserver.cc:460] Webserver started at http://127.0.98.2:36791/ using document root <none> and password file <none>
I20260928 21:14:45.416661 15435 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:45.416754 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:45.420423 15435 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:14:45.422600 15452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.423832 15435 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:14:45.423978 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "0c8825d2f0d84d919c4496ca4f013b74"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "3384c29610fe84e7feaad23d81a5aa46"
server_key_iv: "c0d3ef3cacca8bf52fa98a6b45f49e11"
server_key_version: "encryptionkey@0"
I20260928 21:14:45.424566 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:45.447410 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:45.450416 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:45.450639 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:45.451246 15435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:45.452112 15435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:45.452292 15435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.452421 15435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:45.452477 15435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.461342 15435 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37111
I20260928 21:14:45.461373 15565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37111 every 8 connection(s)
I20260928 21:14:45.462360 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:14:45.466557 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.464763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:38697)
0928 21:14:45.465335 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.465846 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":113,"threads_started":1}
W20260928 21:14:45.467108 15566 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40065 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40065: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
I20260928 21:14:45.467839 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.467606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:58571)
0928 21:14:45.467720 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.467786 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:14:45.468387 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15435
I20260928 21:14:45.468544 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.468294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:49309)
0928 21:14:45.468424 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.468482 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 21:14:45.468734 15566 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:14:45.468911 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:34141
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40065
--webserver_port=36865
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38877
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:14:45.578356 15569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:14:45.578646 15569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:14:45.578691 15569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:14:45.582852 15569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:14:45.582917 15569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:14:45.583030 15569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:14:45.588434 15569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38877
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:34141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=36865
--tserver_master_addrs=127.0.98.62:40065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:14:45.589524 15569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:14:45.590528 15569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:14:45.600656 15574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:45.600747 15569 server_base.cc:1034] running on GCE node
W20260928 21:14:45.600656 15577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:14:45.600696 15575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:14:45.601334 15569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:14:45.602012 15569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:14:45.603193 15569 hybrid_clock.cc:648] HybridClock initialized: now 1790630085603154 us; error 51 us; skew 500 ppm
Sep 28 21:14:45 dist-test-slave-v7xz krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:14:45.605981 15569 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:14:45.606906 15569 webserver.cc:460] Webserver started at http://127.0.98.3:36865/ using document root <none> and password file <none>
I20260928 21:14:45.607472 15569 fs_manager.cc:359] Metadata directory not provided
I20260928 21:14:45.607546 15569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:14:45.611368 15569 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:14:45.613754 15584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.615140 15569 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:14:45.615265 15569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be"
format_stamp: "Formatted at 2026-09-28 21:14:43 on dist-test-slave-v7xz"
server_key: "b2a9cb6252460ec78032998be15891c7"
server_key_iv: "c173984acd3707d400c6be6c12833513"
server_key_version: "encryptionkey@0"
I20260928 21:14:45.615851 15569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:14:45.630856 15569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:14:45.634253 15569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:14:45.634464 15569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:14:45.635076 15569 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:14:45.635910 15569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:14:45.635982 15569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.636053 15569 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:14:45.636094 15569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:14:45.645500 15569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34141
I20260928 21:14:45.645522 15697 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34141 every 8 connection(s)
I20260928 21:14:45.646517 15569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:14:45.650498 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.648738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:58599)
0928 21:14:45.649276 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.649783 (+ 507us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":105,"threads_started":1}
W20260928 21:14:45.651029 15698 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40065 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:40065: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
I20260928 21:14:45.651629 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.651389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:44795)
0928 21:14:45.651495 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.651543 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260928 21:14:45.652287 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:45.652101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:54657)
0928 21:14:45.652198 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:45.652239 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260928 21:14:45.652498 15698 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:14:45.654968 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15569
I20260928 21:14:46.340176 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.339735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:49279)
0928 21:14:46.339982 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:46.340043 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":76,"threads_started":1}
I20260928 21:14:46.469846 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.469355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:58333)
0928 21:14:46.469682 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:46.469737 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":122,"threads_started":1}
I20260928 21:14:46.515964 15704 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:46.516148 15704 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:46.517251 15704 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
Sep 28 21:14:46 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 28 21:14:46 dist-test-slave-v7xz krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630085, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 21:14:46.534438 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.517464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34141 (local address 127.0.98.1:48043)
0928 21:14:46.517694 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:46.517710 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:46.517822 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:46.520026 (+ 2204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:46.520032 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:46.520791 (+ 759us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:46.521752 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:46.521780 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.523796 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.523800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:46.525458 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:46.525470 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.526568 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.526571 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:46.526882 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:46.527668 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:46.527684 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:46.529548 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:46.531894 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:46.531901 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:46.531915 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:46.532282 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:46.532704 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:46.532707 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:46.532708 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:46.532923 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:46.533684 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:46.533690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:46.533916 (+ 226us) client_negotiation.cc:769] Sending connection context
0928 21:14:46.534068 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 21:14:46.534304 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":414,"spinlock_wait_cycles":6272}
I20260928 21:14:46.534718 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.517635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48043 (local address 127.0.98.3:34141)
0928 21:14:46.518115 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:46.518133 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:46.518180 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:14:46.519069 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:46.519078 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:46.519542 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:46.519884 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:46.521980 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.523612 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.525637 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.526386 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.526568 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:46.529799 (+ 3231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:46.529807 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:46.529824 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:46.529856 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:46.531764 (+ 1908us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:46.532409 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:46.532415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:46.532421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:46.532585 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:46.533039 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:46.533043 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:46.533045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:46.533566 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:46.533701 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:46.534202 (+ 501us) server_negotiation.cc:299] Negotiation successful
0928 21:14:46.534444 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":85,"threads_started":1}
I20260928 21:14:46.535107 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.517499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37111 (local address 127.0.98.1:51071)
0928 21:14:46.517807 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:46.517821 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:14:46.517917 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:14:46.520030 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:14:46.520033 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:14:46.520791 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:14:46.521752 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:46.521780 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.523796 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.523800 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:46.525459 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:14:46.525470 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.526536 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.526545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:14:46.526881 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:46.527804 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:14:46.527816 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:14:46.529847 (+ 2031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:14:46.532042 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:46.532045 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:46.532046 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:46.532300 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:46.532672 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:14:46.532677 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:14:46.532679 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:14:46.532906 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:14:46.534608 (+ 1702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:14:46.534611 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:14:46.534699 (+ 88us) client_negotiation.cc:769] Sending connection context
0928 21:14:46.534796 (+ 97us) client_negotiation.cc:240] Negotiation successful
0928 21:14:46.534949 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"mutex_wait_us":396,"spinlock_wait_cycles":4736,"thread_start_us":76,"threads_started":1}
I20260928 21:14:46.535835 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.517658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51071 (local address 127.0.98.2:37111)
0928 21:14:46.518131 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0928 21:14:46.518162 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0928 21:14:46.518192 (+ 30us) server_negotiation.cc:407] Connection header received
0928 21:14:46.519060 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:14:46.519078 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:14:46.519542 (+ 464us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:14:46.519899 (+ 357us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:14:46.521989 (+ 2090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.523612 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.525636 (+ 2024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:14:46.526386 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:14:46.526568 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:14:46.530023 (+ 3455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:14:46.530028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:14:46.530039 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:14:46.530069 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:14:46.531918 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:46.532440 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:46.532444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:46.532449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:46.532574 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:14:46.533019 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:14:46.533038 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:14:46.533040 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:14:46.533565 (+ 525us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:14:46.533701 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0928 21:14:46.535393 (+ 1692us) server_negotiation.cc:299] Negotiation successful
0928 21:14:46.535650 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":115,"threads_started":1}
W20260928 21:14:46.537258 15320 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
W20260928 21:14:46.537714 15318 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
I20260928 21:14:46.537776 15318 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d895b7a9199646e8a73a54a6559e9402; no voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be
I20260928 21:14:46.538074 15704 raft_consensus.cc:2726] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:14:46.653631 15708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:46.653179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:32917)
0928 21:14:46.653435 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:46.653515 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":108,"threads_started":1}
I20260928 21:14:47.341545 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:47.341133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:60971)
0928 21:14:47.341392 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:47.341450 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":80,"threads_started":1}
I20260928 21:14:47.470996 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:47.470593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:54249)
0928 21:14:47.470852 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:47.470908 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260928 21:14:47.654929 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:47.654478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:46967)
0928 21:14:47.654755 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:47.654818 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260928 21:14:48.342907 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:48.342458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:35625)
0928 21:14:48.342745 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:48.342809 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":93,"threads_started":1}
I20260928 21:14:48.380564 15713 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:48.380874 15713 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:48.381409 15713 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
W20260928 21:14:48.382303 15320 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
W20260928 21:14:48.382418 15318 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
I20260928 21:14:48.382537 15320 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d895b7a9199646e8a73a54a6559e9402; no voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be
I20260928 21:14:48.382719 15713 raft_consensus.cc:2726] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:14:48.472332 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:48.471839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:47313)
0928 21:14:48.472128 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:48.472194 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":105,"threads_started":1}
I20260928 21:14:48.656211 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:48.655793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:47955)
0928 21:14:48.656055 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:48.656116 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":79,"threads_started":1}
I20260928 21:14:49.344398 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:49.343941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:51165)
0928 21:14:49.344219 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:49.344281 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":90,"threads_started":1}
I20260928 21:14:49.473666 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:49.473243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:56825)
0928 21:14:49.473507 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:49.473568 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":89,"threads_started":1}
I20260928 21:14:49.657725 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:49.657221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:47469)
0928 21:14:49.657526 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:49.657592 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":100,"threads_started":1}
I20260928 21:14:50.345937 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:50.345500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:52455)
0928 21:14:50.345782 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:50.345840 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260928 21:14:50.474830 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:50.474410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:41607)
0928 21:14:50.474663 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:50.474729 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260928 21:14:50.515208 15721 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:50.515352 15721 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:50.515863 15721 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
W20260928 21:14:50.516774 15318 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
W20260928 21:14:50.516943 15320 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
I20260928 21:14:50.517050 15320 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d895b7a9199646e8a73a54a6559e9402; no voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be
I20260928 21:14:50.517230 15721 raft_consensus.cc:2726] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:14:50.659060 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:50.658615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:33559)
0928 21:14:50.658897 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:50.658962 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":79,"threads_started":1}
I20260928 21:14:51.347198 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:51.346787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:53531)
0928 21:14:51.347047 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:51.347104 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260928 21:14:51.476151 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:51.475681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:43999)
0928 21:14:51.475961 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:51.476026 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":76,"threads_started":1}
I20260928 21:14:51.660478 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:51.660026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:56973)
0928 21:14:51.660319 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:51.660383 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":77,"threads_started":1}
I20260928 21:14:52.348536 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:52.348095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:56627)
0928 21:14:52.348382 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:52.348437 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":96,"threads_started":1}
I20260928 21:14:52.477497 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:52.476989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:48445)
0928 21:14:52.477326 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:52.477395 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":90,"threads_started":1}
I20260928 21:14:52.661860 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:52.661364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:38751)
0928 21:14:52.661692 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:52.661758 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":75,"threads_started":1}
I20260928 21:14:53.349725 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:53.349297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:47685)
0928 21:14:53.349570 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:53.349628 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260928 21:14:53.478880 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:53.478379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:41815)
0928 21:14:53.478676 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:53.478731 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":97,"threads_started":1}
I20260928 21:14:53.663146 15731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:53.662708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:47461)
0928 21:14:53.662987 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:53.663049 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260928 21:14:54.153122 15732 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:54.153265 15732 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:54.153713 15732 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
W20260928 21:14:54.154533 15318 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
W20260928 21:14:54.154635 15320 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
I20260928 21:14:54.154690 15320 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d895b7a9199646e8a73a54a6559e9402; no voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be
I20260928 21:14:54.154824 15732 raft_consensus.cc:2726] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:14:54.350997 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:54.350563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:53251)
0928 21:14:54.350829 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:54.350900 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":82,"threads_started":1}
I20260928 21:14:54.480150 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:54.479647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:42039)
0928 21:14:54.479976 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:54.480036 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":87,"threads_started":1}
I20260928 21:14:54.664376 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:54.663924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:49569)
0928 21:14:54.664217 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:54.664279 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":82,"threads_started":1}
I20260928 21:14:55.352384 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:55.351945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:40187)
0928 21:14:55.352229 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:55.352285 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
I20260928 21:14:55.481467 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:55.481044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:51469)
0928 21:14:55.481313 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:55.481372 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":90,"threads_started":1}
I20260928 21:14:55.665632 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:55.665195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:37867)
0928 21:14:55.665476 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:55.665535 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":90,"threads_started":1}
I20260928 21:14:56.353693 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:56.353229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:42699)
0928 21:14:56.353526 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:56.353587 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
I20260928 21:14:56.482722 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:56.482269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:57145)
0928 21:14:56.482561 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:56.482620 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":107,"threads_started":1}
I20260928 21:14:56.667038 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:56.666531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:41707)
0928 21:14:56.666853 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:56.666920 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":239,"thread_start_us":96,"threads_started":1}
I20260928 21:14:56.941193 15742 raft_consensus.cc:482] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:14:56.941363 15742 raft_consensus.cc:504] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9f4f5bdb42ea4cf8ae2ee51da3a304be" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34141 } } peers { permanent_uuid: "0c8825d2f0d84d919c4496ca4f013b74" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37111 } } peers { permanent_uuid: "d895b7a9199646e8a73a54a6559e9402" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36377 } }
I20260928 21:14:56.941902 15742 leader_election.cc:288] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141), 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111)
W20260928 21:14:56.942677 15320 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9f4f5bdb42ea4cf8ae2ee51da3a304be (127.0.98.3:34141): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
W20260928 21:14:56.942791 15318 leader_election.cc:341] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0c8825d2f0d84d919c4496ca4f013b74 (127.0.98.2:37111): Not found: Tablet not found: 039f7da368584c62a26ea9847a99291d
I20260928 21:14:56.942921 15318 leader_election.cc:302] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d895b7a9199646e8a73a54a6559e9402; no voters: 0c8825d2f0d84d919c4496ca4f013b74, 9f4f5bdb42ea4cf8ae2ee51da3a304be
I20260928 21:14:56.943084 15742 raft_consensus.cc:2726] T 039f7da368584c62a26ea9847a99291d P d895b7a9199646e8a73a54a6559e9402 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260928 21:14:57.355027 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:57.354546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:50945)
0928 21:14:57.354864 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:57.354930 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":92,"threads_started":1}
I20260928 21:14:57.483946 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:57.483472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:43781)
0928 21:14:57.483753 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:57.483817 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":72,"threads_started":1}
I20260928 21:14:57.668327 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:57.667858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:34175)
0928 21:14:57.668147 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:57.668211 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":76,"threads_started":1}
I20260928 21:14:58.356392 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:58.355930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:36521)
0928 21:14:58.356225 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:58.356295 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":90,"threads_started":1}
I20260928 21:14:58.485512 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:58.485067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:39975)
0928 21:14:58.485348 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:58.485412 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260928 21:14:58.669727 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:58.669264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:47631)
0928 21:14:58.669553 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:58.669618 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":77,"threads_started":1}
I20260928 21:14:59.357673 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:59.357227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:58787)
0928 21:14:59.357480 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:59.357538 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":97,"threads_started":1}
I20260928 21:14:59.486884 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:59.486363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:55767)
0928 21:14:59.486654 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:59.486712 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":121,"threads_started":1}
I20260928 21:14:59.671130 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:14:59.670646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:46549)
0928 21:14:59.670958 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0928 21:14:59.671025 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
I20260928 21:15:00.359105 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:00.358653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.1:40591)
0928 21:15:00.358947 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:00.359010 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":72,"threads_started":1}
I20260928 21:15:00.488273 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:00.487818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.2:48369)
0928 21:15:00.488093 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:00.488154 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":75,"threads_started":1}
I20260928 21:15:00.659111 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15268
I20260928 21:15:00.668206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15435
I20260928 21:15:00.672855 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:00.672298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40065 (local address 127.0.98.3:52569)
0928 21:15:00.672602 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:00.672683 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:40065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":83,"threads_started":1}
I20260928 21:15:00.674146 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15569
2026-09-28T21:15:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19268 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:15:00 dist-test-slave-v7xz krb5kdc[15757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:15:00 dist-test-slave-v7xz krb5kdc[15757](info): set up 2 sockets
Sep 28 21:15:00 dist-test-slave-v7xz krb5kdc[15757](info): commencing operation
krb5kdc: starting...
W20260928 21:15:01.721130 392 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:15:01 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:15:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:15:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:15:01.840318 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:15:01.951843 15770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:01.952114 15770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:01.952154 15770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:01.956096 15770 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:15:01.956149 15770 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:15:01.956166 15770 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:15:01.956178 15770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:15:01.956197 15770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:01.956214 15770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:01.956233 15770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:15:01.956251 15770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:15:01.961460 15770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:01.962473 15770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:01.963441 15770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:01.971761 15778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:01.971755 15776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:01.971755 15775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:01.972175 15770 server_base.cc:1034] running on GCE node
I20260928 21:15:01.972687 15770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:01.973817 15770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:01.974982 15770 hybrid_clock.cc:648] HybridClock initialized: now 1790630101974951 us; error 43 us; skew 500 ppm
Sep 28 21:15:01 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630101, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:01.977782 15770 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:15:01.978642 15770 webserver.cc:460] Webserver started at http://127.0.98.62:41487/ using document root <none> and password file <none>
I20260928 21:15:01.979172 15770 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:01.979230 15770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:01.979393 15770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:15:01.981493 15770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "4492b284c1a4479f9839961cefa8f2fc"
format_stamp: "Formatted at 2026-09-28 21:15:01 on dist-test-slave-v7xz"
server_key: "dba91c9875745d92227934120901671a"
server_key_iv: "290836af43161a159f408bcfec1fd531"
server_key_version: "encryptionkey@0"
I20260928 21:15:01.982023 15770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "4492b284c1a4479f9839961cefa8f2fc"
format_stamp: "Formatted at 2026-09-28 21:15:01 on dist-test-slave-v7xz"
server_key: "dba91c9875745d92227934120901671a"
server_key_iv: "290836af43161a159f408bcfec1fd531"
server_key_version: "encryptionkey@0"
I20260928 21:15:01.985634 15770 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:15:01.987980 15785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:01.989202 15770 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:15:01.989328 15770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "4492b284c1a4479f9839961cefa8f2fc"
format_stamp: "Formatted at 2026-09-28 21:15:01 on dist-test-slave-v7xz"
server_key: "dba91c9875745d92227934120901671a"
server_key_iv: "290836af43161a159f408bcfec1fd531"
server_key_version: "encryptionkey@0"
I20260928 21:15:01.989429 15770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:02.009541 15770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:02.012305 15770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:02.012514 15770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:02.019439 15770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41635
I20260928 21:15:02.019722 15837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41635 every 8 connection(s)
I20260928 21:15:02.020434 15770 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:15:02.023193 15838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:02.026481 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15770
I20260928 21:15:02.026582 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:15:02.026904 392 external_mini_cluster.cc:1428] Setting key f18336b25f5e77b808531e38232b4d30
I20260928 21:15:02.029172 15838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap starting.
I20260928 21:15:02.031880 15838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:02.032699 15838 log.cc:824] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:02.034891 15838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: No bootstrap required, opened a new log
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630101, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:15:02.039386 15838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:02.039742 15838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:02.039805 15838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4492b284c1a4479f9839961cefa8f2fc, State: Initialized, Role: FOLLOWER
I20260928 21:15:02.040315 15838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:02.040491 15838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:15:02.040606 15838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:15:02.040738 15838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:02.041924 15838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:02.042220 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.028350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40636 (local address 127.0.98.62:41635)
0928 21:15:02.028778 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.028798 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.028859 (+ 61us) server_negotiation.cc:407] Connection header received
0928 21:15:02.029703 (+ 844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.029727 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.030135 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.030525 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:02.031187 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.032651 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.033520 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.034165 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.034335 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.036724 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:02.036731 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:02.036746 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:02.036776 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:02.038938 (+ 2162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.039365 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.039371 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.039377 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.039582 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.039868 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.039873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.039876 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.040373 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:02.040572 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.040906 (+ 334us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.041212 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":107,"threads_started":1}
I20260928 21:15:02.042275 15838 leader_election.cc:302] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4492b284c1a4479f9839961cefa8f2fc; no voters:
I20260928 21:15:02.042654 15838 leader_election.cc:288] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:15:02.042719 15843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:15:02.042928 15843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 LEADER]: Becoming Leader. State: Replica: 4492b284c1a4479f9839961cefa8f2fc, State: Running, Role: LEADER
I20260928 21:15:02.043289 15843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:02.043994 15838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:15:02.045348 15845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4492b284c1a4479f9839961cefa8f2fc. Latest consensus state: current_term: 1 leader_uuid: "4492b284c1a4479f9839961cefa8f2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260928 21:15:02.045414 15844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4492b284c1a4479f9839961cefa8f2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260928 21:15:02.045571 15845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: This master's current role is: LEADER
I20260928 21:15:02.045576 15844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: This master's current role is: LEADER
I20260928 21:15:02.046542 15853 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:15:02.049265 15853 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:15:02.054978 15853 catalog_manager.cc:1347] Generated new cluster ID: 0fc1c8c263f54974b90cf662912461f2
I20260928 21:15:02.055126 15853 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:15:02.070217 15853 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:15:02.071563 15853 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:15:02.086786 15853 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Generated new TSK 0
I20260928 21:15:02.087692 15853 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:15:02.119410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:15:02.231624 15864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:02.231870 15864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:02.231911 15864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:02.236012 15864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:02.236074 15864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:02.236132 15864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:15:02.241462 15864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:02.242489 15864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:02.243480 15864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:02.253043 15872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:02.253039 15869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:02.253041 15870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:02.253526 15864 server_base.cc:1034] running on GCE node
I20260928 21:15:02.253950 15864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:02.254534 15864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:02.255698 15864 hybrid_clock.cc:648] HybridClock initialized: now 1790630102255668 us; error 41 us; skew 500 ppm
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:02.258380 15864 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:15:02.259189 15864 webserver.cc:460] Webserver started at http://127.0.98.1:41323/ using document root <none> and password file <none>
I20260928 21:15:02.259727 15864 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:02.259785 15864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:02.260011 15864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:15:02.262188 15864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "5460281e919b48ceb5f1b09c0dd3506e"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "5b1623d0c0ddb4b109ea69aec0e7baa3"
server_key_iv: "b89043b0cdab63a2982930ec5b9f4729"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.262748 15864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5460281e919b48ceb5f1b09c0dd3506e"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "5b1623d0c0ddb4b109ea69aec0e7baa3"
server_key_iv: "b89043b0cdab63a2982930ec5b9f4729"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.266523 15864 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:15:02.268723 15879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.269896 15864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:15:02.270033 15864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "5460281e919b48ceb5f1b09c0dd3506e"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "5b1623d0c0ddb4b109ea69aec0e7baa3"
server_key_iv: "b89043b0cdab63a2982930ec5b9f4729"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.270159 15864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:02.295459 15864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:02.298213 15864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:02.298400 15864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:02.298957 15864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:02.299759 15864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:15:02.299826 15864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.299896 15864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:15:02.299948 15864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.308589 15864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46749
I20260928 21:15:02.308629 15992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46749 every 8 connection(s)
I20260928 21:15:02.309513 15864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:15:02.315748 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15864
I20260928 21:15:02.315848 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:15:02.316162 392 external_mini_cluster.cc:1428] Setting key 713c09faeaf79e9b23c04384eacd9089
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:15:02.323606 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.311209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52475 (local address 127.0.98.62:41635)
0928 21:15:02.311342 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.311345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.312147 (+ 802us) server_negotiation.cc:407] Connection header received
0928 21:15:02.313549 (+ 1402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.313552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.313612 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.313686 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:02.314940 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.315485 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.316917 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.317159 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.317278 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.319748 (+ 2470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:02.319753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:02.319757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:02.319788 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:02.321517 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.321996 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.322003 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.322008 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.322099 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.322414 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.322419 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.322423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.322685 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:02.322815 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.323271 (+ 456us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.323412 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:15:02.324276 15995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.311587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:52475)
0928 21:15:02.312006 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.312047 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.313328 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.313854 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.313861 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.314219 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:02.314792 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.314807 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.315615 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.315619 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.316762 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.316771 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.317310 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.317314 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.317492 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.318091 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:02.318107 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:02.319565 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:02.321642 (+ 2077us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:02.321646 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:02.321658 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:02.321892 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:02.322199 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:02.322201 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:02.322202 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:02.322319 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:02.322812 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:02.322817 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:02.323031 (+ 214us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.323187 (+ 156us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.323420 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":112,"threads_started":1}
I20260928 21:15:02.325512 15993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:15:02.325773 15993 heartbeater.cc:464] Registering TS with master...
I20260928 21:15:02.326306 15993 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260928 21:15:02.327977 15802 ts_manager.cc:194] Registered new tserver with Master: 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:15:02.329607 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52475
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:15:02.344403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:15:02.453788 15998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:02.454070 15998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:02.454183 15998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:02.458246 15998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:02.458326 15998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:02.458448 15998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:15:02.463665 15998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:02.464668 15998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:02.465713 15998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:02.475098 16006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:02.475100 16004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:02.475135 16003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:02.475461 15998 server_base.cc:1034] running on GCE node
I20260928 21:15:02.475920 15998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:02.476598 15998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:02.477957 15998 hybrid_clock.cc:648] HybridClock initialized: now 1790630102477914 us; error 40 us; skew 500 ppm
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:02.480636 15998 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:15:02.481487 15998 webserver.cc:460] Webserver started at http://127.0.98.2:32779/ using document root <none> and password file <none>
I20260928 21:15:02.481988 15998 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:02.482043 15998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:02.482197 15998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:15:02.484084 15998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "2318442a84d2460fa2441a9b8ff20ff9"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "4f39061570291d5a356130d17a6e4603"
server_key_iv: "bdb24e879f80d55e18e18c1101277948"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.484583 15998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "2318442a84d2460fa2441a9b8ff20ff9"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "4f39061570291d5a356130d17a6e4603"
server_key_iv: "bdb24e879f80d55e18e18c1101277948"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.488139 15998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:15:02.490536 16013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.491726 15998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:15:02.491819 15998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "2318442a84d2460fa2441a9b8ff20ff9"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "4f39061570291d5a356130d17a6e4603"
server_key_iv: "bdb24e879f80d55e18e18c1101277948"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.491953 15998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:02.515787 15998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:02.518721 15998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:02.518872 15998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:02.519403 15998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:02.520205 15998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:15:02.520254 15998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.520293 15998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:15:02.520356 15998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.529397 15998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41497
I20260928 21:15:02.529420 16126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41497 every 8 connection(s)
I20260928 21:15:02.530313 15998 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:15:02.530719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 15998
I20260928 21:15:02.530860 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:15:02.531200 392 external_mini_cluster.cc:1428] Setting key 65132c3f5a0337701f4b1afb50446c29
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:15:02.544458 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.532248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59279 (local address 127.0.98.62:41635)
0928 21:15:02.532383 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.532388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.533429 (+ 1041us) server_negotiation.cc:407] Connection header received
0928 21:15:02.534735 (+ 1306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.534738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.534791 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.534871 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:02.536012 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.536635 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.537809 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.538048 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.538167 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.540707 (+ 2540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:02.540710 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:02.540712 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:02.540734 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:02.542223 (+ 1489us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.542747 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.542753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.542755 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.542836 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.543177 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.543180 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.543182 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.543427 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:02.543543 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.544082 (+ 539us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.544223 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:15:02.545534 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.532751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:59279)
0928 21:15:02.533259 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.533312 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.534519 (+ 1207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.535038 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.535045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.535404 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:02.535872 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.535885 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.536768 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.536771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.537683 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.537693 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.538199 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.538203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.538365 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.538965 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:02.538981 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:02.540561 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:02.542359 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:02.542365 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:02.542378 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:02.542643 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:02.542928 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:02.542933 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:02.542935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:02.543073 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:02.543573 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:02.543578 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:02.543805 (+ 227us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.543997 (+ 192us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.544247 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":141,"threads_started":1}
I20260928 21:15:02.546850 16127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260928 21:15:02.547128 16127 heartbeater.cc:464] Registering TS with master...
I20260928 21:15:02.547752 16127 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260928 21:15:02.548936 15802 ts_manager.cc:194] Registered new tserver with Master: 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497)
I20260928 21:15:02.549770 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:15:02.557786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:15:02.672251 16132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:02.672583 16132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:02.672633 16132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:02.676735 16132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:02.676791 16132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:02.676914 16132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:15:02.682102 16132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:02.683147 16132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:02.684079 16132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:02.693197 16137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:02.693257 16140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:02.693528 16132 server_base.cc:1034] running on GCE node
W20260928 21:15:02.693224 16138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:02.693987 16132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:02.694630 16132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:02.695840 16132 hybrid_clock.cc:648] HybridClock initialized: now 1790630102695806 us; error 53 us; skew 500 ppm
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:02.698787 16132 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:15:02.699642 16132 webserver.cc:460] Webserver started at http://127.0.98.3:44373/ using document root <none> and password file <none>
I20260928 21:15:02.700214 16132 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:02.700304 16132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:02.700534 16132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:15:02.702647 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "ed7109ce995249fda574868ff9e62b64"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "8e2d4fe291481d38036c57209807f307"
server_key_iv: "6779d7e974e2940d646e2af53230fd37"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.703222 16132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ed7109ce995249fda574868ff9e62b64"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "8e2d4fe291481d38036c57209807f307"
server_key_iv: "6779d7e974e2940d646e2af53230fd37"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.706985 16132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260928 21:15:02.709458 16147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.710665 16132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:15:02.710834 16132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "ed7109ce995249fda574868ff9e62b64"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "8e2d4fe291481d38036c57209807f307"
server_key_iv: "6779d7e974e2940d646e2af53230fd37"
server_key_version: "encryptionkey@0"
I20260928 21:15:02.710950 16132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:02.739144 16132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:02.742559 16132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:02.742764 16132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:02.743366 16132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:02.744308 16132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:15:02.744378 16132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.744446 16132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:15:02.744498 16132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:02.753614 16132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41845
I20260928 21:15:02.753639 16260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41845 every 8 connection(s)
I20260928 21:15:02.754549 16132 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
Sep 28 21:15:02 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630102, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:15:02.764228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16132
I20260928 21:15:02.764370 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:15:02.764683 392 external_mini_cluster.cc:1428] Setting key a40765c8bb62371229467d0ab22dd92d
I20260928 21:15:02.768805 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.756531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41455 (local address 127.0.98.62:41635)
0928 21:15:02.756692 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.756696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.757566 (+ 870us) server_negotiation.cc:407] Connection header received
0928 21:15:02.758850 (+ 1284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.758853 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.758933 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.759039 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:02.760189 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.760720 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.761899 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.762117 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.762232 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.764938 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:02.764941 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:02.764943 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:02.764966 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:02.766791 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.767310 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.767314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.767316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.767405 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.767745 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.767749 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.767750 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.767932 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:02.768034 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.768525 (+ 491us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.768642 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 21:15:02.769695 16263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.756939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:41455)
0928 21:15:02.757397 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.757440 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.758635 (+ 1195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.759213 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.759220 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.759579 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:02.760037 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.760051 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.760879 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.760882 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.761760 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.761769 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.762266 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.762270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.762449 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.763067 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:02.763083 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:02.764705 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:02.766907 (+ 2202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:02.766913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:02.766928 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:02.767182 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:02.767499 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:02.767503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:02.767505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:02.767643 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:02.768045 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:02.768052 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:02.768291 (+ 239us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.768469 (+ 178us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.768781 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260928 21:15:02.770853 16261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260928 21:15:02.771052 16261 heartbeater.cc:464] Registering TS with master...
I20260928 21:15:02.771497 16261 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260928 21:15:02.772439 15802 ts_manager.cc:194] Registered new tserver with Master: ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845)
I20260928 21:15:02.773028 15802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41455
I20260928 21:15:02.778594 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:15:02.787775 15841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.780516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.98.62:41635)
0928 21:15:02.780651 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.780656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.780866 (+ 210us) server_negotiation.cc:407] Connection header received
0928 21:15:02.780984 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.780986 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.781024 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.781091 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:02.781889 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.782374 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.783159 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.783361 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.783436 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.785013 (+ 1577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:02.785020 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:02.785023 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:02.785050 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:02.786308 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.786684 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.786687 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.786688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.786760 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:02.786983 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:02.786985 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:02.786987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:02.787188 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:02.787301 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.787486 (+ 185us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.787577 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260928 21:15:02.790589 15802 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40650:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:15:02.793594 15802 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:15:02.807924 16276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.802811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41497 (local address 127.0.0.1:49818)
0928 21:15:02.803283 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.803295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.803385 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.803949 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.803952 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.803969 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:02.804051 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.804057 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.805068 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.805071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.806502 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.806513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.807328 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.807332 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.807433 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.807470 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.807577 (+ 107us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.807732 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"mutex_wait_us":123,"spinlock_wait_cycles":13312,"thread_start_us":93,"threads_started":1}
I20260928 21:15:02.808137 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.802353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.0.98.3:41845)
0928 21:15:02.802693 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.802704 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.802725 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:15:02.802797 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.802802 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.803029 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.803540 (+ 511us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:02.803717 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.804818 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.806635 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.807453 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.807558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.807693 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.807901 (+ 208us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.808007 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":157,"threads_started":1}
I20260928 21:15:02.808413 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.803220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.0.98.2:41497)
0928 21:15:02.803625 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.803634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.803651 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:15:02.803707 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.803710 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.803826 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.803968 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:02.804141 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.804949 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.806665 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.807184 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.807319 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.807477 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.807986 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.808266 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":89,"threads_started":1}
I20260928 21:15:02.808543 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.802113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.0.98.1:46749)
0928 21:15:02.802399 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.802418 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.802688 (+ 270us) server_negotiation.cc:407] Connection header received
0928 21:15:02.802777 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.802783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.803012 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.803165 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:02.803615 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.804819 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.806408 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.807226 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.807336 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.807492 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.808243 (+ 751us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.808351 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":98,"threads_started":1}
I20260928 21:15:02.808120 16273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.801716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.0.1:34888)
0928 21:15:02.802470 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.802484 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.802582 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.803347 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.803350 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.803383 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:02.803484 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.803492 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.805047 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.805050 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.806285 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.806294 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.807558 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.807562 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.807746 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.807788 (+ 42us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.807879 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.807952 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"mutex_wait_us":157,"spinlock_wait_cycles":1792,"thread_start_us":94,"threads_started":1}
I20260928 21:15:02.808130 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.801649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41845 (local address 127.0.0.1:38678)
0928 21:15:02.801993 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.802038 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.802158 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.803208 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.803215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.803241 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:02.803395 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.803402 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.804957 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.804960 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.806502 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.806513 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.807563 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.807570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.807735 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.807775 (+ 40us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.807844 (+ 69us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.807905 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":129,"thread_start_us":102,"threads_started":1}
I20260928 21:15:02.811630 15927 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e (DEFAULT_TABLE table=test-table [id=8c414928f48341bbb6ec5bafcfaee379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:15:02.811579 16061 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e (DEFAULT_TABLE table=test-table [id=8c414928f48341bbb6ec5bafcfaee379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:15:02.812011 16195 tablet_service.cc:1467] Processing CreateTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e (DEFAULT_TABLE table=test-table [id=8c414928f48341bbb6ec5bafcfaee379]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:15:02.812633 16061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf301dade3d4a3aa626d4eaa9b5342e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:02.812639 15927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf301dade3d4a3aa626d4eaa9b5342e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:02.812928 16195 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bf301dade3d4a3aa626d4eaa9b5342e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:02.819075 16278 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap starting.
I20260928 21:15:02.820158 16279 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Bootstrap starting.
I20260928 21:15:02.820585 16280 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Bootstrap starting.
I20260928 21:15:02.821208 16278 tablet_bootstrap.cc:654] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:02.822244 16278 log.cc:824] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:02.822662 16279 tablet_bootstrap.cc:654] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:02.823091 16280 tablet_bootstrap.cc:654] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:02.823922 16280 log.cc:824] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:02.823922 16279 log.cc:824] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:02.824121 16278 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: No bootstrap required, opened a new log
I20260928 21:15:02.824352 16278 ts_tablet_manager.cc:1397] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:15:02.825861 16280 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: No bootstrap required, opened a new log
I20260928 21:15:02.825861 16279 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: No bootstrap required, opened a new log
I20260928 21:15:02.826041 16280 ts_tablet_manager.cc:1397] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:15:02.826041 16279 ts_tablet_manager.cc:1397] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:15:02.827162 16278 raft_consensus.cc:348] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.827376 16278 raft_consensus.cc:374] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:02.827416 16278 raft_consensus.cc:729] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5460281e919b48ceb5f1b09c0dd3506e, State: Initialized, Role: FOLLOWER
I20260928 21:15:02.827836 16278 consensus_queue.cc:260] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.828433 15993 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260928 21:15:02.828758 16278 ts_tablet_manager.cc:1428] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260928 21:15:02.829706 16280 raft_consensus.cc:348] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.829946 16280 raft_consensus.cc:374] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:02.830032 16280 raft_consensus.cc:729] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2318442a84d2460fa2441a9b8ff20ff9, State: Initialized, Role: FOLLOWER
I20260928 21:15:02.830487 16280 consensus_queue.cc:260] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.830499 16279 raft_consensus.cc:348] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.830763 16279 raft_consensus.cc:374] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:02.830880 16279 raft_consensus.cc:729] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7109ce995249fda574868ff9e62b64, State: Initialized, Role: FOLLOWER
I20260928 21:15:02.831138 16127 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260928 21:15:02.831462 16280 ts_tablet_manager.cc:1428] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:15:02.831396 16279 consensus_queue.cc:260] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.832034 16261 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260928 21:15:02.832293 16279 ts_tablet_manager.cc:1428] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:15:02.885814 16285 raft_consensus.cc:482] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:15:02.886082 16285 raft_consensus.cc:504] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.887338 16285 leader_election.cc:288] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845), 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:02.891430 16274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.887715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51221 (local address 127.0.98.1:46749)
0928 21:15:02.887861 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.887865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.887880 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:15:02.888009 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.888012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.888065 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.888153 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:02.888442 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.889036 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.890369 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.890786 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.890907 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.890969 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.891223 (+ 254us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.891296 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:15:02.891438 16275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.887809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42763 (local address 127.0.98.3:41845)
0928 21:15:02.887949 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.887953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.888183 (+ 230us) server_negotiation.cc:407] Connection header received
0928 21:15:02.888347 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.888350 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.888420 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.888507 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:02.888762 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.889260 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.890377 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.890792 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.890908 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.890972 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.891220 (+ 248us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.891295 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:15:02.891436 16129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.887618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.98.2:51221)
0928 21:15:02.887757 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.887774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.887883 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.888166 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.888169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.888187 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:02.888324 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.888330 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.889145 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.889151 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.890229 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.890239 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.890914 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.890917 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.891028 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.891099 (+ 71us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.891204 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.891295 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":38,"spinlock_wait_cycles":36224}
I20260928 21:15:02.891453 16287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.887618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41845 (local address 127.0.98.2:42763)
0928 21:15:02.888088 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.888099 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:02.888242 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:02.888518 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:02.888521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:02.888535 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:02.888654 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.888660 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.889373 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.889378 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.890256 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:02.890265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.890929 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.890933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:02.891064 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.891101 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:15:02.891202 (+ 101us) client_negotiation.cc:240] Negotiation successful
0928 21:15:02.891296 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":417,"mutex_wait_us":180,"thread_start_us":129,"threads_started":1}
I20260928 21:15:02.892145 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf301dade3d4a3aa626d4eaa9b5342e" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5460281e919b48ceb5f1b09c0dd3506e" is_pre_election: true
I20260928 21:15:02.892210 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf301dade3d4a3aa626d4eaa9b5342e" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7109ce995249fda574868ff9e62b64" is_pre_election: true
I20260928 21:15:02.892483 15947 raft_consensus.cc:2454] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2318442a84d2460fa2441a9b8ff20ff9 in term 0.
I20260928 21:15:02.892531 16215 raft_consensus.cc:2454] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2318442a84d2460fa2441a9b8ff20ff9 in term 0.
I20260928 21:15:02.893090 16017 leader_election.cc:302] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2318442a84d2460fa2441a9b8ff20ff9, 5460281e919b48ceb5f1b09c0dd3506e, ed7109ce995249fda574868ff9e62b64; no voters:
I20260928 21:15:02.893394 16285 raft_consensus.cc:2781] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:15:02.893496 16285 raft_consensus.cc:482] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:15:02.893539 16285 raft_consensus.cc:3037] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:02.894616 16285 raft_consensus.cc:504] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.895030 16285 leader_election.cc:288] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 1 election: Requested vote from peers ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845), 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:02.895404 16215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf301dade3d4a3aa626d4eaa9b5342e" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed7109ce995249fda574868ff9e62b64"
I20260928 21:15:02.895454 15947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bf301dade3d4a3aa626d4eaa9b5342e" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5460281e919b48ceb5f1b09c0dd3506e"
I20260928 21:15:02.895571 16215 raft_consensus.cc:3037] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:02.895593 15947 raft_consensus.cc:3037] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:02.896792 16215 raft_consensus.cc:2454] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2318442a84d2460fa2441a9b8ff20ff9 in term 1.
I20260928 21:15:02.896798 15947 raft_consensus.cc:2454] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2318442a84d2460fa2441a9b8ff20ff9 in term 1.
I20260928 21:15:02.897158 16017 leader_election.cc:302] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2318442a84d2460fa2441a9b8ff20ff9, ed7109ce995249fda574868ff9e62b64; no voters:
I20260928 21:15:02.897346 16285 raft_consensus.cc:2781] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:15:02.897585 16285 raft_consensus.cc:686] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 LEADER]: Becoming Leader. State: Replica: 2318442a84d2460fa2441a9b8ff20ff9, State: Running, Role: LEADER
I20260928 21:15:02.897989 16285 consensus_queue.cc:237] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:02.901540 15801 catalog_manager.cc:5495] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "2318442a84d2460fa2441a9b8ff20ff9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } health_report { overall_health: HEALTHY } } }
I20260928 21:15:02.933213 16277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.929493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49826 (local address 127.0.98.2:41497)
0928 21:15:02.929616 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:02.929620 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:02.929632 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:15:02.929733 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:02.929736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:02.929813 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:02.929936 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:15:02.930265 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.930789 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.931675 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:02.931893 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:02.931978 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:02.932269 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:15:02.932692 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:15:02.932786 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:02.932975 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:15:02.933031 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260928 21:15:02.939251 15947 raft_consensus.cc:1261] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Refusing update from remote peer 2318442a84d2460fa2441a9b8ff20ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:15:02.939354 16215 raft_consensus.cc:1261] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Refusing update from remote peer 2318442a84d2460fa2441a9b8ff20ff9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:15:02.940068 16285 consensus_queue.cc:1035] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:15:02.940261 16288 consensus_queue.cc:1035] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:15:02.948088 16293 mvcc.cc:205] Tried to move back new op lower bound from 7334420901632626688 to 7334420901474332672. Current Snapshot: MvccSnapshot[applied={T|T < 7334420901632626688}]
I20260928 21:15:02.957734 15801 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:15:02.957896 15801 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40650:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:15:02.960270 15801 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Sending DeleteTablet for 3 replicas of tablet 8bf301dade3d4a3aa626d4eaa9b5342e
I20260928 21:15:02.961094 16195 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38678
I20260928 21:15:02.961078 16061 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49818
I20260928 21:15:02.961611 15927 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34888
I20260928 21:15:02.961769 16301 tablet_replica.cc:330] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: stopping tablet replica
I20260928 21:15:02.961971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15864
I20260928 21:15:02.963801 16302 tablet_replica.cc:330] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: stopping tablet replica
I20260928 21:15:02.964107 16302 raft_consensus.cc:2229] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:15:02.964490 16302 raft_consensus.cc:2258] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:15:02.964826 16301 raft_consensus.cc:2229] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:15:02.965234 16301 raft_consensus.cc:2258] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:15:02.966197 16302 ts_tablet_manager.cc:1905] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:15:02.966655 16301 ts_tablet_manager.cc:1905] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:15:02.969190 16302 ts_tablet_manager.cc:1918] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:15:02.969364 16301 ts_tablet_manager.cc:1918] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:15:02.969389 16302 log.cc:1196] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/8bf301dade3d4a3aa626d4eaa9b5342e
I20260928 21:15:02.969542 16301 log.cc:1196] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/8bf301dade3d4a3aa626d4eaa9b5342e
I20260928 21:15:02.969739 16302 ts_tablet_manager.cc:1939] T 8bf301dade3d4a3aa626d4eaa9b5342e P 2318442a84d2460fa2441a9b8ff20ff9: Deleting consensus metadata
I20260928 21:15:02.969952 16301 ts_tablet_manager.cc:1939] T 8bf301dade3d4a3aa626d4eaa9b5342e P ed7109ce995249fda574868ff9e62b64: Deleting consensus metadata
W20260928 21:15:02.970180 15787 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46749 (error 108)
I20260928 21:15:02.971009 15788 catalog_manager.cc:4841] TS 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497): tablet 8bf301dade3d4a3aa626d4eaa9b5342e (table test-table [id=8c414928f48341bbb6ec5bafcfaee379]) successfully deleted
I20260928 21:15:02.971014 15789 catalog_manager.cc:4841] TS ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845): tablet 8bf301dade3d4a3aa626d4eaa9b5342e (table test-table [id=8c414928f48341bbb6ec5bafcfaee379]) successfully deleted
I20260928 21:15:02.971285 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15998
I20260928 21:15:02.972770 16272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:02.972415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.0.1:34890)
0928 21:15:02.972578 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:02.972665 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 21:15:02.976974 15787 catalog_manager.cc:4584] TS 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bf301dade3d4a3aa626d4eaa9b5342e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46749: connect: Connection refused (error 111)
I20260928 21:15:02.977474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 16132
I20260928 21:15:02.983075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 15770
I20260928 21:15:22.989641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--webserver_interface=127.0.98.62
--webserver_port=41487
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:15:23.106961 16305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:23.107245 16305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:23.107287 16305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:23.111280 16305 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:15:23.111331 16305 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:15:23.111357 16305 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:15:23.111371 16305 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:15:23.111390 16305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:23.111409 16305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:23.111424 16305 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:15:23.111445 16305 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:15:23.116643 16305 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:23.117733 16305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:23.118714 16305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:23.132335 16311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:23.132407 16314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:23.132329 16312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:23.132642 16305 server_base.cc:1034] running on GCE node
I20260928 21:15:23.133365 16305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:23.134621 16305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:23.135828 16305 hybrid_clock.cc:648] HybridClock initialized: now 1790630123135808 us; error 31 us; skew 500 ppm
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:23.138829 16305 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:15:23.139879 16305 webserver.cc:460] Webserver started at http://127.0.98.62:41487/ using document root <none> and password file <none>
I20260928 21:15:23.140498 16305 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:23.140619 16305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:23.144650 16305 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:15:23.146983 16321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.148257 16305 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:15:23.148402 16305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "4492b284c1a4479f9839961cefa8f2fc"
format_stamp: "Formatted at 2026-09-28 21:15:01 on dist-test-slave-v7xz"
server_key: "dba91c9875745d92227934120901671a"
server_key_iv: "290836af43161a159f408bcfec1fd531"
server_key_version: "encryptionkey@0"
I20260928 21:15:23.149082 16305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:23.172132 16305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:23.175513 16305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:23.175674 16305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:23.183410 16305 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41635
I20260928 21:15:23.183702 16373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41635 every 8 connection(s)
I20260928 21:15:23.184561 16305 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:15:23.186322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16305
I20260928 21:15:23.186790 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46749
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41635
--webserver_port=41323
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:15:23.189201 16374 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:15:23.190987 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap starting.
I20260928 21:15:23.208245 16374 log.cc:824] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:23.216660 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:15:23.217109 16374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap complete.
I20260928 21:15:23.221098 16374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:23.221513 16374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4492b284c1a4479f9839961cefa8f2fc, State: Initialized, Role: FOLLOWER
I20260928 21:15:23.221989 16374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:23.222136 16374 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:15:23.222223 16374 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:15:23.222345 16374 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:15:23.224171 16374 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:23.224501 16374 leader_election.cc:302] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4492b284c1a4479f9839961cefa8f2fc; no voters:
I20260928 21:15:23.224835 16374 leader_election.cc:288] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [CANDIDATE]: Term 2 election: Requested vote from peers
I20260928 21:15:23.224936 16378 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:15:23.225162 16378 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 LEADER]: Becoming Leader. State: Replica: 4492b284c1a4479f9839961cefa8f2fc, State: Running, Role: LEADER
I20260928 21:15:23.225517 16378 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:23.226090 16374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:15:23.226457 16378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4492b284c1a4479f9839961cefa8f2fc. Latest consensus state: current_term: 2 leader_uuid: "4492b284c1a4479f9839961cefa8f2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260928 21:15:23.226615 16378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: This master's current role is: LEADER
I20260928 21:15:23.226915 16379 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4492b284c1a4479f9839961cefa8f2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260928 21:15:23.227087 16379 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: This master's current role is: LEADER
I20260928 21:15:23.227010 16382 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:15:23.232290 16382 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:15:23.233186 16382 catalog_manager.cc:1259] Loaded cluster ID: 0fc1c8c263f54974b90cf662912461f2
I20260928 21:15:23.233249 16382 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:15:23.235550 16382 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:15:23.236220 16382 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Loaded TSK: 0
I20260928 21:15:23.253196 16382 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Generated new TSK 1
I20260928 21:15:23.253480 16382 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:15:23.305562 16376 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:23.305871 16376 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:23.305956 16376 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:23.310021 16376 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:23.310083 16376 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:23.310143 16376 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:15:23.315506 16376 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41323
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16376
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:23.316510 16376 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:23.317523 16376 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:23.327045 16404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:23.327068 16401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:23.327314 16376 server_base.cc:1034] running on GCE node
W20260928 21:15:23.327108 16402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:23.327951 16376 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:23.328709 16376 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:23.329854 16376 hybrid_clock.cc:648] HybridClock initialized: now 1790630123329838 us; error 38 us; skew 500 ppm
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:23.333156 16376 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:15:23.334053 16376 webserver.cc:460] Webserver started at http://127.0.98.1:41323/ using document root <none> and password file <none>
I20260928 21:15:23.334604 16376 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:23.334714 16376 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:23.338706 16376 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:15:23.341048 16411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.342391 16376 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:15:23.342543 16376 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "5460281e919b48ceb5f1b09c0dd3506e"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "5b1623d0c0ddb4b109ea69aec0e7baa3"
server_key_iv: "b89043b0cdab63a2982930ec5b9f4729"
server_key_version: "encryptionkey@0"
I20260928 21:15:23.343185 16376 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:23.355826 16376 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:23.359426 16376 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:23.359652 16376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:23.360289 16376 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:23.361563 16418 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:15:23.366636 16376 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:15:23.366719 16376 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.001s
I20260928 21:15:23.366804 16376 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:15:23.368523 16376 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:15:23.368597 16376 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:15:23.368906 16418 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap starting.
I20260928 21:15:23.379148 16376 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46749
I20260928 21:15:23.379247 16525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46749 every 8 connection(s)
I20260928 21:15:23.380125 16376 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:15:23.383440 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16376
I20260928 21:15:23.383985 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41497
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41635
--webserver_port=32779
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:15:23.390149 16418 log.cc:824] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Log is configured to *not* fsync() on all Append() calls
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:15:23.398308 16418 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:15:23.398813 16418 tablet_bootstrap.cc:492] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap complete.
I20260928 21:15:23.399653 16528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.382230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:59673)
0928 21:15:23.382680 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.382732 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.383586 (+ 854us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.384805 (+ 1219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.384813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.385271 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:23.385938 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.385960 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.388044 (+ 2084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.388046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.388993 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.389002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.389968 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.389975 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.390217 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.390972 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:23.391001 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:23.393010 (+ 2009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:23.395822 (+ 2812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.395828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.395841 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.396184 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.396680 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.396683 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.396686 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.396894 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.397758 (+ 864us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:23.397764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:23.398089 (+ 325us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.398296 (+ 207us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.398668 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":135,"threads_started":1}
I20260928 21:15:23.399854 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.382269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59673 (local address 127.0.98.62:41635)
0928 21:15:23.382737 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.382754 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.382833 (+ 79us) server_negotiation.cc:407] Connection header received
0928 21:15:23.384003 (+ 1170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.384014 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.384411 (+ 397us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.384745 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:23.386155 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.387868 (+ 1713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.389112 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.389806 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.389976 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.393248 (+ 3272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:23.393258 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:23.393279 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:23.393320 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:23.395686 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.396331 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.396339 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.396348 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.396533 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.397048 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.397055 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.397059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.397614 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:23.397797 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.398533 (+ 736us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.398847 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":124,"threads_started":1}
I20260928 21:15:23.400230 16418 ts_tablet_manager.cc:1397] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Time spent bootstrapping tablet: real 0.032s user 0.014s sys 0.015s
I20260928 21:15:23.402707 16526 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260928 21:15:23.402946 16526 heartbeater.cc:464] Registering TS with master...
I20260928 21:15:23.403469 16526 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260928 21:15:23.403179 16418 raft_consensus.cc:348] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:23.403637 16418 raft_consensus.cc:729] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5460281e919b48ceb5f1b09c0dd3506e, State: Initialized, Role: FOLLOWER
I20260928 21:15:23.404100 16418 consensus_queue.cc:260] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } }
I20260928 21:15:23.405145 16418 ts_tablet_manager.cc:1428] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Time spent starting tablet: real 0.005s user 0.005s sys 0.001s
I20260928 21:15:23.406607 16337 ts_manager.cc:194] Registered new tserver with Master: 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:23.410002 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59673
I20260928 21:15:23.412460 16526 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:15:23.420055 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.409364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.0.1:44656)
0928 21:15:23.410055 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.410099 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.410290 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.410984 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.410988 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.411426 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:23.411562 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.411571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.412960 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.412964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.413611 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.413621 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.414357 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.414373 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.414496 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.415045 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:23.415056 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:23.416353 (+ 1297us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:23.418308 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.418316 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.418321 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.418553 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.418853 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.418857 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.418859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.418953 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.419403 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:23.419409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:23.419533 (+ 124us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.419669 (+ 136us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.419862 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":124,"threads_started":1}
I20260928 21:15:23.420055 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.409453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44656 (local address 127.0.98.1:46749)
0928 21:15:23.409810 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.409821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.410185 (+ 364us) server_negotiation.cc:407] Connection header received
0928 21:15:23.410471 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.410477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.410672 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.410833 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:23.411693 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.412813 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.413740 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.414242 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.414367 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.416499 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:23.416508 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:23.416520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:23.416551 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:23.418174 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.418670 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.418676 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.418677 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.418759 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.419046 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.419051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.419053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.419287 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:23.419428 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.419759 (+ 331us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.419912 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267,"thread_start_us":97,"threads_started":1}
I20260928 21:15:23.421077 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bf301dade3d4a3aa626d4eaa9b5342e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44656
I20260928 21:15:23.421869 16535 tablet_replica.cc:330] stopping tablet replica
I20260928 21:15:23.422158 16535 raft_consensus.cc:2229] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:15:23.422399 16535 raft_consensus.cc:2258] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:15:23.423311 16535 ts_tablet_manager.cc:1905] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:15:23.427037 16535 ts_tablet_manager.cc:1918] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:15:23.427160 16535 log.cc:1196] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal/wals/8bf301dade3d4a3aa626d4eaa9b5342e
I20260928 21:15:23.427484 16535 ts_tablet_manager.cc:1939] T 8bf301dade3d4a3aa626d4eaa9b5342e P 5460281e919b48ceb5f1b09c0dd3506e: Deleting consensus metadata
I20260928 21:15:23.429093 16323 catalog_manager.cc:4841] TS 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749): tablet 8bf301dade3d4a3aa626d4eaa9b5342e (table test-table [id=8c414928f48341bbb6ec5bafcfaee379]) successfully deleted
W20260928 21:15:23.502928 16530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:23.503198 16530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:23.503245 16530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:23.507146 16530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:23.507208 16530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:23.507266 16530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:15:23.512356 16530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32779
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:23.513381 16530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:23.514372 16530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:23.524179 16541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:23.524180 16543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:23.524365 16530 server_base.cc:1034] running on GCE node
W20260928 21:15:23.524271 16540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:23.524979 16530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:23.525604 16530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:23.526793 16530 hybrid_clock.cc:648] HybridClock initialized: now 1790630123526777 us; error 23 us; skew 500 ppm
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:23.529524 16530 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:15:23.530441 16530 webserver.cc:460] Webserver started at http://127.0.98.2:32779/ using document root <none> and password file <none>
I20260928 21:15:23.530979 16530 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:23.531049 16530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:23.534862 16530 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:15:23.536764 16550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.537962 16530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:15:23.538079 16530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "2318442a84d2460fa2441a9b8ff20ff9"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "4f39061570291d5a356130d17a6e4603"
server_key_iv: "bdb24e879f80d55e18e18c1101277948"
server_key_version: "encryptionkey@0"
I20260928 21:15:23.538589 16530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:23.552157 16530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:23.555229 16530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:23.555438 16530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:23.556059 16530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:23.556912 16530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:15:23.556958 16530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.557029 16530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:15:23.557067 16530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.565579 16530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41497
I20260928 21:15:23.565640 16663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41497 every 8 connection(s)
I20260928 21:15:23.566622 16530 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:15:23.573217 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16530
I20260928 21:15:23.573658 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41845
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41635
--webserver_port=44373
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:15:23.581086 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.568342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35889 (local address 127.0.98.62:41635)
0928 21:15:23.568495 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.568498 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.569344 (+ 846us) server_negotiation.cc:407] Connection header received
0928 21:15:23.570620 (+ 1276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.570623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.570679 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.570772 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:23.571951 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.572578 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.573867 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.574110 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.574171 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.576716 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:23.576723 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:23.576729 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:23.576765 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:23.578716 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.579304 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.579310 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.579314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.579420 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.579742 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.579748 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.579752 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.580063 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:23.580173 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.580669 (+ 496us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.580848 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:15:23.581807 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.568718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:35889)
0928 21:15:23.569194 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.569236 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.570411 (+ 1175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.570962 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.570969 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.571345 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:23.571811 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.571826 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.572701 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.572704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.573706 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.573715 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.574346 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.574348 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.574490 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.575102 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:23.575118 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:23.576542 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:23.578851 (+ 2309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.578856 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.578918 (+ 62us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.579202 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.579520 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.579522 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.579523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.579647 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.580181 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:23.580187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:23.580431 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.580585 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.580886 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":159,"threads_started":1}
I20260928 21:15:23.583043 16664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260928 21:15:23.583272 16664 heartbeater.cc:464] Registering TS with master...
I20260928 21:15:23.583752 16664 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260928 21:15:23.584764 16337 ts_manager.cc:194] Registered new tserver with Master: 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497)
I20260928 21:15:23.585376 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35889
W20260928 21:15:23.685392 16667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:23.685688 16667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:23.685761 16667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:23.689742 16667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:23.689821 16667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:23.689921 16667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:15:23.695149 16667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44373
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:23.696240 16667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:23.697338 16667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:23.706765 16675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:23.706773 16673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:23.706773 16672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:23.706943 16667 server_base.cc:1034] running on GCE node
I20260928 21:15:23.707460 16667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:23.708102 16667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:23.709348 16667 hybrid_clock.cc:648] HybridClock initialized: now 1790630123709330 us; error 30 us; skew 500 ppm
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:23.712265 16667 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:15:23.713218 16667 webserver.cc:460] Webserver started at http://127.0.98.3:44373/ using document root <none> and password file <none>
I20260928 21:15:23.713792 16667 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:23.713903 16667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:23.717775 16667 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260928 21:15:23.719882 16682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.721401 16667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:15:23.721541 16667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "ed7109ce995249fda574868ff9e62b64"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "8e2d4fe291481d38036c57209807f307"
server_key_iv: "6779d7e974e2940d646e2af53230fd37"
server_key_version: "encryptionkey@0"
I20260928 21:15:23.722110 16667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:23.736300 16667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:23.739341 16667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:23.739542 16667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:23.740281 16667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:23.741233 16667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:15:23.741325 16667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.741412 16667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:15:23.741462 16667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:23.750900 16667 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41845
I20260928 21:15:23.750964 16795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41845 every 8 connection(s)
I20260928 21:15:23.752013 16667 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:15:23.760339 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16667
Sep 28 21:15:23 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:15:23.766326 16529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.753977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57197 (local address 127.0.98.62:41635)
0928 21:15:23.754112 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.754115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.755016 (+ 901us) server_negotiation.cc:407] Connection header received
0928 21:15:23.756312 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.756315 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.756363 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.756448 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:23.757674 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.758189 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.759280 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.759499 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.759602 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.762300 (+ 2698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:23.762304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:23.762307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:23.762334 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:23.763987 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.764514 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.764518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.764521 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.764611 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.765009 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.765013 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.765014 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.765285 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:23.765390 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.765982 (+ 592us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.766119 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:15:23.767030 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.754385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:57197)
0928 21:15:23.754844 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.754896 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.756106 (+ 1210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.756604 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.756610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.757051 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:23.757527 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.757540 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.758294 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.758296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.759154 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.759162 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.759638 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.759640 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.759772 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.760376 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:23.760392 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:23.762096 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:23.764137 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.764142 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.764154 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.764409 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.764696 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:23.764699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:23.764701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:23.764900 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:23.765407 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:23.765414 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:23.765719 (+ 305us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.765873 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.766123 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":136,"threads_started":1}
I20260928 21:15:23.768324 16796 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41635
I20260928 21:15:23.768579 16796 heartbeater.cc:464] Registering TS with master...
I20260928 21:15:23.769157 16796 heartbeater.cc:511] Master 127.0.98.62:41635 requested a full tablet report, sending...
I20260928 21:15:23.770315 16337 ts_manager.cc:194] Registered new tserver with Master: ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845)
I20260928 21:15:23.770735 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.761076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43310 (local address 127.0.98.62:41635)
0928 21:15:23.761306 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.761310 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.761510 (+ 200us) server_negotiation.cc:407] Connection header received
0928 21:15:23.761661 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.761665 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.761711 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.761798 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:23.762665 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.763419 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.764556 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.764885 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.765006 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.767391 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:23.767394 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:23.767396 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:23.767415 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:23.769012 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.769474 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.769477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.769478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.769552 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.769855 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.769858 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.769859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.770066 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:23.770217 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.770432 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.770548 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260928 21:15:23.771026 16337 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57197
I20260928 21:15:23.772538 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:15:23.776211 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.773179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.0.98.62:41635)
0928 21:15:23.773286 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.773289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.773537 (+ 248us) server_negotiation.cc:407] Connection header received
0928 21:15:23.773642 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.773644 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.773725 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.773799 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:15:23.773937 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.774402 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.775278 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.775487 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.775561 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.775821 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:15:23.775936 (+ 115us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0928 21:15:23.776122 (+ 186us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43326: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 21:15:23.776288 16799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43326: token expired
W20260928 21:15:23.776706 16264 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:41635)
I20260928 21:15:23.783713 16799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.777074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.0.98.62:41635)
0928 21:15:23.777178 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.777181 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.777235 (+ 54us) server_negotiation.cc:407] Connection header received
0928 21:15:23.777332 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.777334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.777364 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.777424 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:23.777977 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.778420 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.779299 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.779488 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.779590 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.780914 (+ 1324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:23.780917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:23.780919 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:23.780939 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:23.782354 (+ 1415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.782755 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.782758 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.782759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.782837 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:23.783055 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:23.783057 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:23.783058 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:23.783200 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:23.783292 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.783530 (+ 238us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.783612 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:15:23.786196 16337 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43328:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:15:23.789460 16337 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:15:23.797124 16460 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2d714d8cc344d7917e65345ec83c7d (DEFAULT_TABLE table=test-table [id=4ae84e8c7a544be0be3064a3a38eee8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:15:23.797799 16460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2d714d8cc344d7917e65345ec83c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:23.800702 16418 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap starting.
I20260928 21:15:23.801074 16806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.796492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41497 (local address 127.0.0.1:40444)
0928 21:15:23.796876 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.796890 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.796982 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.797638 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.797640 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.797657 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:23.797762 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.797768 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.798750 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.798756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.799832 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.799841 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.800642 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.800646 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.800726 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.800767 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.800851 (+ 84us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.800930 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":33,"thread_start_us":117,"threads_started":1}
I20260928 21:15:23.801160 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.796667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40444 (local address 127.0.98.2:41497)
0928 21:15:23.797189 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.797197 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.797215 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:15:23.797274 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.797278 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.797502 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.797665 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:23.797866 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.798630 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.799957 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.800518 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.800641 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.800795 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.800909 (+ 114us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.801002 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":95,"threads_started":1}
I20260928 21:15:23.801656 16418 tablet_bootstrap.cc:654] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:23.803139 16418 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: No bootstrap required, opened a new log
I20260928 21:15:23.803221 16418 ts_tablet_manager.cc:1397] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260928 21:15:23.803586 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.795991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41845 (local address 127.0.0.1:45300)
0928 21:15:23.796155 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.796172 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.796256 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.797431 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.797434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.797452 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:23.797563 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.797570 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.799066 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.799071 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.800210 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.800224 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.803158 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.803162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.803258 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.803278 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.803369 (+ 91us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.803448 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":1}
I20260928 21:15:23.803650 16418 raft_consensus.cc:348] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.803757 16418 raft_consensus.cc:374] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:23.803792 16418 raft_consensus.cc:729] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5460281e919b48ceb5f1b09c0dd3506e, State: Initialized, Role: FOLLOWER
I20260928 21:15:23.803848 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.796398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45300 (local address 127.0.98.3:41845)
0928 21:15:23.796901 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.796917 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.796943 (+ 26us) server_negotiation.cc:407] Connection header received
0928 21:15:23.797021 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.797027 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.797253 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.797726 (+ 473us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:23.797906 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.798929 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.800374 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.803030 (+ 2656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.803158 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.803297 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.803557 (+ 260us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.803679 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":161,"threads_started":1}
I20260928 21:15:23.803913 16418 consensus_queue.cc:260] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.804625 16418 ts_tablet_manager.cc:1428] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260928 21:15:23.804329 16598 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2d714d8cc344d7917e65345ec83c7d (DEFAULT_TABLE table=test-table [id=4ae84e8c7a544be0be3064a3a38eee8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:15:23.805474 16598 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2d714d8cc344d7917e65345ec83c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:23.806495 16730 tablet_service.cc:1467] Processing CreateTablet for tablet 4e2d714d8cc344d7917e65345ec83c7d (DEFAULT_TABLE table=test-table [id=4ae84e8c7a544be0be3064a3a38eee8f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:15:23.807621 16730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e2d714d8cc344d7917e65345ec83c7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:15:23.812368 16809 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Bootstrap starting.
I20260928 21:15:23.813822 16810 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Bootstrap starting.
I20260928 21:15:23.814795 16809 tablet_bootstrap.cc:654] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:23.815507 16809 log.cc:824] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:23.815804 16810 tablet_bootstrap.cc:654] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Neither blocks nor log segments found. Creating new log.
I20260928 21:15:23.816921 16810 log.cc:824] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:23.817461 16809 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: No bootstrap required, opened a new log
I20260928 21:15:23.817669 16809 ts_tablet_manager.cc:1397] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:15:23.818550 16810 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: No bootstrap required, opened a new log
I20260928 21:15:23.818743 16810 ts_tablet_manager.cc:1397] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260928 21:15:23.820947 16809 raft_consensus.cc:348] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.821187 16809 raft_consensus.cc:374] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:23.821265 16809 raft_consensus.cc:729] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2318442a84d2460fa2441a9b8ff20ff9, State: Initialized, Role: FOLLOWER
I20260928 21:15:23.821703 16809 consensus_queue.cc:260] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.821883 16810 raft_consensus.cc:348] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.822130 16810 raft_consensus.cc:374] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:15:23.822190 16810 raft_consensus.cc:729] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7109ce995249fda574868ff9e62b64, State: Initialized, Role: FOLLOWER
I20260928 21:15:23.822324 16664 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260928 21:15:23.822650 16809 ts_tablet_manager.cc:1428] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:15:23.822577 16810 consensus_queue.cc:260] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.823228 16796 heartbeater.cc:503] Master 127.0.98.62:41635 was elected leader, sending a full tablet report...
I20260928 21:15:23.823518 16810 ts_tablet_manager.cc:1428] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260928 21:15:23.881425 16527 tablet.cc:2369] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:15:23.977407 16814 raft_consensus.cc:482] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:15:23.977813 16814 raft_consensus.cc:504] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.979270 16814 leader_election.cc:288] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497), 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:23.982914 16807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.979737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57981 (local address 127.0.98.2:41497)
0928 21:15:23.979868 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.979873 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.979892 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:15:23.979939 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.979941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.979978 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.980075 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:23.980376 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.980939 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.981958 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.982369 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.982491 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.982557 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.982739 (+ 182us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.982804 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:15:23.982914 16798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.979471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41497 (local address 127.0.98.3:57981)
0928 21:15:23.979584 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.979599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.979697 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.980124 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.980127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.980145 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:23.980261 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.980266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.981040 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.981044 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.981833 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.981842 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.982492 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.982495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.982612 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.982630 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.982710 (+ 80us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.982781 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":2}
I20260928 21:15:23.983135 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.979628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39681 (local address 127.0.98.1:46749)
0928 21:15:23.979937 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:23.979942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:23.980005 (+ 63us) server_negotiation.cc:407] Connection header received
0928 21:15:23.980146 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:23.980149 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:23.980231 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:23.980345 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:15:23.980609 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.981158 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.982202 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.982626 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.982752 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.982916 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:23.982955 (+ 39us) server_negotiation.cc:299] Negotiation successful
0928 21:15:23.983023 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260928 21:15:23.983135 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:23.979563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.98.3:39681)
0928 21:15:23.979923 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:23.979938 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:23.980046 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:23.980353 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:23.980356 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:23.980370 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:15:23.980495 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.980500 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.981271 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.981276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.982068 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:23.982076 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:23.982762 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:23.982765 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:23.982856 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:23.982875 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:15:23.982937 (+ 62us) client_negotiation.cc:240] Negotiation successful
0928 21:15:23.983000 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":21,"thread_start_us":130,"threads_started":1}
I20260928 21:15:23.983721 16480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "ed7109ce995249fda574868ff9e62b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5460281e919b48ceb5f1b09c0dd3506e" is_pre_election: true
I20260928 21:15:23.983940 16618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "ed7109ce995249fda574868ff9e62b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2318442a84d2460fa2441a9b8ff20ff9" is_pre_election: true
I20260928 21:15:23.984002 16480 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed7109ce995249fda574868ff9e62b64 in term 0.
I20260928 21:15:23.984226 16618 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed7109ce995249fda574868ff9e62b64 in term 0.
I20260928 21:15:23.984431 16684 leader_election.cc:302] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5460281e919b48ceb5f1b09c0dd3506e, ed7109ce995249fda574868ff9e62b64; no voters:
I20260928 21:15:23.984735 16814 raft_consensus.cc:2781] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:15:23.984824 16814 raft_consensus.cc:482] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:15:23.984908 16814 raft_consensus.cc:3037] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:23.985893 16814 raft_consensus.cc:504] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.986218 16814 leader_election.cc:288] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [CANDIDATE]: Term 1 election: Requested vote from peers 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497), 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:23.986567 16618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "ed7109ce995249fda574868ff9e62b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2318442a84d2460fa2441a9b8ff20ff9"
I20260928 21:15:23.986567 16480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "ed7109ce995249fda574868ff9e62b64" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5460281e919b48ceb5f1b09c0dd3506e"
I20260928 21:15:23.986685 16480 raft_consensus.cc:3037] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:23.986696 16618 raft_consensus.cc:3037] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:15:23.987887 16480 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed7109ce995249fda574868ff9e62b64 in term 1.
I20260928 21:15:23.987887 16618 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed7109ce995249fda574868ff9e62b64 in term 1.
I20260928 21:15:23.988215 16685 leader_election.cc:302] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2318442a84d2460fa2441a9b8ff20ff9, ed7109ce995249fda574868ff9e62b64; no voters:
I20260928 21:15:23.988404 16814 raft_consensus.cc:2781] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:15:23.988622 16814 raft_consensus.cc:686] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 LEADER]: Becoming Leader. State: Replica: ed7109ce995249fda574868ff9e62b64, State: Running, Role: LEADER
I20260928 21:15:23.988984 16814 consensus_queue.cc:237] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:23.991935 16338 catalog_manager.cc:5495] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed7109ce995249fda574868ff9e62b64 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ed7109ce995249fda574868ff9e62b64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } health_report { overall_health: UNKNOWN } } }
W20260928 21:15:24.003152 16797 tablet.cc:2369] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:15:24.038223 16805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:24.034418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45308 (local address 127.0.98.3:41845)
0928 21:15:24.034560 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:24.034564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:24.034579 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:15:24.034715 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:24.034717 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:24.034773 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:24.034886 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:15:24.035126 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:24.035621 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:24.036668 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:24.036945 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:24.037055 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:24.037371 (+ 316us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:15:24.037809 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:15:24.037940 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:24.037998 (+ 58us) server_negotiation.cc:299] Negotiation successful
0928 21:15:24.038056 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:15:24.044190 16480 raft_consensus.cc:1261] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Refusing update from remote peer ed7109ce995249fda574868ff9e62b64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:15:24.044207 16618 raft_consensus.cc:1261] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Refusing update from remote peer ed7109ce995249fda574868ff9e62b64: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:15:24.044965 16814 consensus_queue.cc:1035] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:15:24.045186 16817 consensus_queue.cc:1035] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:15:24.050922 16820 mvcc.cc:205] Tried to move back new op lower bound from 7334420988077346816 to 7334420987861676032. Current Snapshot: MvccSnapshot[applied={T|T < 7334420988077346816}]
I20260928 21:15:24.052917 16821 mvcc.cc:205] Tried to move back new op lower bound from 7334420988077346816 to 7334420987861676032. Current Snapshot: MvccSnapshot[applied={T|T < 7334420988077346816}]
I20260928 21:15:24.059851 16338 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:15:24.060026 16338 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43328:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:15:24.062270 16338 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Sending DeleteTablet for 3 replicas of tablet 4e2d714d8cc344d7917e65345ec83c7d
I20260928 21:15:24.062973 16460 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2d714d8cc344d7917e65345ec83c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44656
I20260928 21:15:24.062943 16730 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2d714d8cc344d7917e65345ec83c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45300
I20260928 21:15:24.062983 16598 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e2d714d8cc344d7917e65345ec83c7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:15:24 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40444
I20260928 21:15:24.063403 16829 tablet_replica.cc:330] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: stopping tablet replica
I20260928 21:15:24.063550 16829 raft_consensus.cc:2229] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:15:24.063707 16829 raft_consensus.cc:2258] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:15:24.063742 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 16376
I20260928 21:15:24.068500 16831 tablet_replica.cc:330] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: stopping tablet replica
I20260928 21:15:24.068820 16830 tablet_replica.cc:330] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: stopping tablet replica
I20260928 21:15:24.068861 16831 raft_consensus.cc:2229] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:15:24.069181 16831 raft_consensus.cc:2258] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:15:24.069218 16830 raft_consensus.cc:2229] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:15:24.069685 16830 raft_consensus.cc:2258] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260928 21:15:24.070256 16323 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:46749 (error 108)
I20260928 21:15:24.070377 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 16530
I20260928 21:15:24.076026 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 16667
I20260928 21:15:24.076570 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:24.076230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.0.1:44660)
0928 21:15:24.076408 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:24.076479 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20260928 21:15:24.076858 16323 catalog_manager.cc:4584] TS 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e2d714d8cc344d7917e65345ec83c7d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:46749: connect: Connection refused (error 111)
I20260928 21:15:24.078356 16533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:24.078102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41497 (local address 127.0.0.1:40460)
0928 21:15:24.078229 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:24.078289 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:41497: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260928 21:15:24.084009 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 16305
I20260928 21:15:44.090243 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--webserver_interface=127.0.98.62
--webserver_port=41487
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:15:44.208392 16833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:44.208683 16833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:44.208725 16833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:44.213196 16833 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260928 21:15:44.213266 16833 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260928 21:15:44.213284 16833 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260928 21:15:44.213297 16833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:15:44.213348 16833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:44.213385 16833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:44.213406 16833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:15:44.213439 16833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:15:44.219429 16833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:44.220618 16833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:44.221755 16833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:15:44.230652 16833 server_base.cc:1034] running on GCE node
W20260928 21:15:44.230597 16838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.230623 16839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.230573 16841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:44.231418 16833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:44.232532 16833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:44.233716 16833 hybrid_clock.cc:648] HybridClock initialized: now 1790630144233667 us; error 65 us; skew 500 ppm
Sep 28 21:15:44 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:44.236650 16833 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:15:44.237622 16833 webserver.cc:460] Webserver started at http://127.0.98.62:41487/ using document root <none> and password file <none>
I20260928 21:15:44.238204 16833 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:44.238304 16833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:44.242141 16833 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.002s
I20260928 21:15:44.244369 16848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:44.245808 16833 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:15:44.245981 16833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "4492b284c1a4479f9839961cefa8f2fc"
format_stamp: "Formatted at 2026-09-28 21:15:01 on dist-test-slave-v7xz"
server_key: "dba91c9875745d92227934120901671a"
server_key_iv: "290836af43161a159f408bcfec1fd531"
server_key_version: "encryptionkey@0"
I20260928 21:15:44.246681 16833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:44.260694 16833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:44.264140 16833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:44.264355 16833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:44.271610 16833 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41635
I20260928 21:15:44.271927 16900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41635 every 8 connection(s)
I20260928 21:15:44.272768 16833 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:15:44.277042 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16833
I20260928 21:15:44.277320 16901 sys_catalog.cc:263] Verifying existing consensus state
I20260928 21:15:44.277549 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:46749
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41635
--webserver_port=41323
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:15:44.279341 16901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap starting.
I20260928 21:15:44.297266 16901 log.cc:824] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:44.308179 16901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:15:44.308578 16901 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Bootstrap complete.
I20260928 21:15:44.312573 16901 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:44.313009 16901 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4492b284c1a4479f9839961cefa8f2fc, State: Initialized, Role: FOLLOWER
I20260928 21:15:44.313541 16901 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:44.313656 16901 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:15:44.313740 16901 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:15:44.313855 16901 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 2 FOLLOWER]: Advancing to term 3
I20260928 21:15:44.315723 16901 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:44.316064 16901 leader_election.cc:302] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4492b284c1a4479f9839961cefa8f2fc; no voters:
I20260928 21:15:44.316396 16901 leader_election.cc:288] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [CANDIDATE]: Term 3 election: Requested vote from peers
I20260928 21:15:44.316488 16905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 3 FOLLOWER]: Leader election won for term 3
I20260928 21:15:44.316713 16905 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [term 3 LEADER]: Becoming Leader. State: Replica: 4492b284c1a4479f9839961cefa8f2fc, State: Running, Role: LEADER
I20260928 21:15:44.317085 16905 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } }
I20260928 21:15:44.317647 16901 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:15:44.318476 16906 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4492b284c1a4479f9839961cefa8f2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260928 21:15:44.318594 16906 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: This master's current role is: LEADER
I20260928 21:15:44.318972 16907 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4492b284c1a4479f9839961cefa8f2fc. Latest consensus state: current_term: 3 leader_uuid: "4492b284c1a4479f9839961cefa8f2fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4492b284c1a4479f9839961cefa8f2fc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41635 } } }
I20260928 21:15:44.319104 16907 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc [sys.catalog]: This master's current role is: LEADER
I20260928 21:15:44.319964 16914 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:15:44.324348 16914 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:15:44.324988 16914 catalog_manager.cc:1259] Loaded cluster ID: 0fc1c8c263f54974b90cf662912461f2
I20260928 21:15:44.325057 16914 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:15:44.326828 16914 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:15:44.327430 16914 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Loaded TSK: 0
I20260928 21:15:44.327517 16914 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4492b284c1a4479f9839961cefa8f2fc: Loaded TSK: 1
F20260928 21:15:44.328100 16914 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7d47bee9ad google::LogMessage::Fail() at ??:0
@ 0x7f7d47bf188a google::LogMessage::SendToLog() at ??:0
@ 0x7f7d47bee4de google::LogMessage::Flush() at ??:0
@ 0x7f7d47befa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7d4a8b218d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7d4a8aaf0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7d525537bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7d5252d99c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7d5252e178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f7d5256fdf8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7d5258e82b std::function<>::operator()() at ??:0
@ 0x7f7d5252db4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7d5252ef3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7d5252b579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7d5256f640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7d49e403d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7d49e40c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7d49e423f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7d49e31512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7d4b4756db start_thread at ??:0
@ 0x7f7d466ec61f clone at ??:0
W20260928 21:15:44.427991 16903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:44.428233 16903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:44.428277 16903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:44.432420 16903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:44.432483 16903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:44.432576 16903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:15:44.438153 16903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:46749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41323
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16903
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:44.439244 16903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:44.440260 16903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:44.450178 16948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.450178 16949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.450404 16951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:44.450330 16903 server_base.cc:1034] running on GCE node
I20260928 21:15:44.450917 16903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:44.451610 16903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:44.452786 16903 hybrid_clock.cc:648] HybridClock initialized: now 1790630144452773 us; error 31 us; skew 500 ppm
Sep 28 21:15:44 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:44.456130 16903 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:15:44.457147 16903 webserver.cc:460] Webserver started at http://127.0.98.1:41323/ using document root <none> and password file <none>
I20260928 21:15:44.457721 16903 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:44.457783 16903 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:44.461935 16903 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:15:44.464397 16958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:44.465865 16903 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:15:44.466010 16903 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "5460281e919b48ceb5f1b09c0dd3506e"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "5b1623d0c0ddb4b109ea69aec0e7baa3"
server_key_iv: "b89043b0cdab63a2982930ec5b9f4729"
server_key_version: "encryptionkey@0"
I20260928 21:15:44.466593 16903 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:44.483372 16903 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:44.486970 16903 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:44.487196 16903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:44.487833 16903 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:44.489028 16965 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:15:44.495122 16903 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:15:44.495258 16903 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20260928 21:15:44.495348 16903 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:15:44.497184 16903 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:15:44.497262 16903 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.002s
I20260928 21:15:44.497599 16965 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap starting.
I20260928 21:15:44.508018 16903 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46749
I20260928 21:15:44.509123 16903 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:15:44.509078 17072 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46749 every 8 connection(s)
I20260928 21:15:44.515064 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 16903
I20260928 21:15:44.515604 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41497
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41635
--webserver_port=32779
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:15:44.520756 16965 log.cc:824] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:44.531749 16965 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:15:44.532249 16965 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Bootstrap complete.
I20260928 21:15:44.533725 16965 ts_tablet_manager.cc:1397] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Time spent bootstrapping tablet: real 0.036s user 0.022s sys 0.012s
I20260928 21:15:44.537242 16965 raft_consensus.cc:348] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:44.537820 16965 raft_consensus.cc:729] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5460281e919b48ceb5f1b09c0dd3506e, State: Initialized, Role: FOLLOWER
I20260928 21:15:44.538431 16965 consensus_queue.cc:260] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:44.539551 16965 ts_tablet_manager.cc:1428] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260928 21:15:44.561126 17075 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:15:44.511514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:38105)
0928 21:15:44.512092 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.512184 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:44.513447 (+ 1263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:44.560222 (+ 46775us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":126,"threads_started":1}
W20260928 21:15:44.561728 17073 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41635 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41635: Client connection negotiation failed: client connection to 127.0.98.62:41635: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260928 21:15:44.562578 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.562310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:39329)
0928 21:15:44.562449 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.562518 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 21:15:44.563364 17075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.563122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:56499)
0928 21:15:44.563254 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.563302 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 21:15:44.563628 17073 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260928 21:15:44.645931 17076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:44.646256 17076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:44.646350 17076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:44.650491 17076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:44.650612 17076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:44.650754 17076 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:15:44.656430 17076 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=32779
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:44.657660 17076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:44.658803 17076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:44.669029 17086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.668998 17083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.669010 17084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:44.669224 17076 server_base.cc:1034] running on GCE node
I20260928 21:15:44.669741 17076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:44.670450 17076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:44.671648 17076 hybrid_clock.cc:648] HybridClock initialized: now 1790630144671606 us; error 52 us; skew 500 ppm
Sep 28 21:15:44 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:44.674582 17076 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:15:44.675496 17076 webserver.cc:460] Webserver started at http://127.0.98.2:32779/ using document root <none> and password file <none>
I20260928 21:15:44.676030 17076 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:44.676088 17076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:44.680007 17076 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:15:44.682461 17093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:44.684020 17076 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260928 21:15:44.684182 17076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "2318442a84d2460fa2441a9b8ff20ff9"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "4f39061570291d5a356130d17a6e4603"
server_key_iv: "bdb24e879f80d55e18e18c1101277948"
server_key_version: "encryptionkey@0"
I20260928 21:15:44.684792 17076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:44.698995 17076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:44.702776 17076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:44.702998 17076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:44.703655 17076 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:44.704794 17100 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:15:44.709205 17076 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:15:44.709297 17076 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 21:15:44.709393 17076 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:15:44.711220 17076 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:15:44.711277 17076 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260928 21:15:44.711622 17100 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Bootstrap starting.
I20260928 21:15:44.718497 17100 log.cc:824] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:44.722121 17076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41497
I20260928 21:15:44.722195 17208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41497 every 8 connection(s)
I20260928 21:15:44.723044 17076 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:15:44.725651 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17076
I20260928 21:15:44.726183 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:41845
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41635
--webserver_port=44373
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40495
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:15:44.726683 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.725146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:40149)
0928 21:15:44.725655 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.725787 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":179,"threads_started":1}
W20260928 21:15:44.727154 17209 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41635 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41635: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
I20260928 21:15:44.727828 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.727612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:43775)
0928 21:15:44.727715 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.727766 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260928 21:15:44.728336 17100 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:15:44.728524 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.728247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:46337)
0928 21:15:44.728403 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.728459 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260928 21:15:44.728739 17209 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:15:44.728900 17100 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Bootstrap complete.
I20260928 21:15:44.729264 17100 ts_tablet_manager.cc:1397] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.001s
I20260928 21:15:44.732597 17100 raft_consensus.cc:348] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:44.734606 17100 raft_consensus.cc:729] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2318442a84d2460fa2441a9b8ff20ff9, State: Initialized, Role: FOLLOWER
I20260928 21:15:44.735255 17100 consensus_queue.cc:260] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:44.736583 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.736318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:51259)
0928 21:15:44.736449 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.736509 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 21:15:44.736649 17100 ts_tablet_manager.cc:1428] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:15:44.737463 17211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.737220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:36029)
0928 21:15:44.737344 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.737406 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 21:15:44.846271 17212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:15:44.846619 17212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:15:44.846719 17212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:15:44.850841 17212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:15:44.850908 17212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:15:44.851001 17212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:15:44.856458 17212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40495
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:41845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44373
--tserver_master_addrs=127.0.98.62:41635
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:15:44.857582 17212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:15:44.858630 17212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:15:44.868487 17218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:15:44.868477 17219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:44.868618 17212 server_base.cc:1034] running on GCE node
W20260928 21:15:44.868793 17221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:15:44.869300 17212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:15:44.870052 17212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:15:44.871312 17212 hybrid_clock.cc:648] HybridClock initialized: now 1790630144871283 us; error 51 us; skew 500 ppm
Sep 28 21:15:44 dist-test-slave-v7xz krb5kdc[15757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:15:44.874289 17212 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:15:44.875128 17212 webserver.cc:460] Webserver started at http://127.0.98.3:44373/ using document root <none> and password file <none>
I20260928 21:15:44.875654 17212 fs_manager.cc:359] Metadata directory not provided
I20260928 21:15:44.875713 17212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:15:44.879526 17212 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:15:44.882155 17228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:15:44.883430 17212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:15:44.883530 17212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "ed7109ce995249fda574868ff9e62b64"
format_stamp: "Formatted at 2026-09-28 21:15:02 on dist-test-slave-v7xz"
server_key: "8e2d4fe291481d38036c57209807f307"
server_key_iv: "6779d7e974e2940d646e2af53230fd37"
server_key_version: "encryptionkey@0"
I20260928 21:15:44.884047 17212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:15:44.902313 17212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:15:44.905797 17212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:15:44.905961 17212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:15:44.906512 17212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:15:44.907567 17235 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260928 21:15:44.911880 17212 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260928 21:15:44.911975 17212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260928 21:15:44.912060 17212 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260928 21:15:44.913866 17212 ts_tablet_manager.cc:610] Registered 1 tablets
I20260928 21:15:44.913952 17212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260928 21:15:44.914196 17235 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Bootstrap starting.
I20260928 21:15:44.920490 17235 log.cc:824] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Log is configured to *not* fsync() on all Append() calls
I20260928 21:15:44.924362 17212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:41845
I20260928 21:15:44.924481 17343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:41845 every 8 connection(s)
I20260928 21:15:44.925364 17212 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:15:44.927688 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17212
I20260928 21:15:44.928934 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.927385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:55335)
0928 21:15:44.927875 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.928012 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":143,"threads_started":1}
W20260928 21:15:44.929373 17344 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41635 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41635: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
I20260928 21:15:44.929749 17235 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260928 21:15:44.930008 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.929759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:59509)
0928 21:15:44.929905 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.929957 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260928 21:15:44.930220 17235 tablet_bootstrap.cc:492] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Bootstrap complete.
I20260928 21:15:44.930621 17235 ts_tablet_manager.cc:1397] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.000s
I20260928 21:15:44.930706 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.930518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:57957)
0928 21:15:44.930618 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.930659 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:15:44.930866 17344 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260928 21:15:44.933547 17235 raft_consensus.cc:348] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:44.934007 17235 raft_consensus.cc:729] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7109ce995249fda574868ff9e62b64, State: Initialized, Role: FOLLOWER
I20260928 21:15:44.934504 17235 consensus_queue.cc:260] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:44.935503 17235 ts_tablet_manager.cc:1428] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:15:44.935639 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.935402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:58013)
0928 21:15:44.935509 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.935556 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:15:44.936276 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:44.936067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:41145)
0928 21:15:44.936176 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:44.936227 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260928 21:15:45.564548 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:45.564164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:41983)
0928 21:15:45.564396 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:45.564447 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":72,"threads_started":1}
I20260928 21:15:45.738494 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:45.738109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:54359)
0928 21:15:45.738342 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:45.738392 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":83,"threads_started":1}
I20260928 21:15:45.937448 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:45.937055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:53847)
0928 21:15:45.937300 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:45.937358 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":71,"threads_started":1}
I20260928 21:15:46.126856 17352 raft_consensus.cc:482] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:15:46.127099 17352 raft_consensus.cc:504] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:46.128325 17352 leader_election.cc:288] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845), 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497)
Sep 28 21:15:46 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260928 21:15:46.143646 17357 raft_consensus.cc:482] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
Sep 28 21:15:46 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 21:15:46.143862 17357 raft_consensus.cc:504] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:46.146849 17357 leader_election.cc:288] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845), 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:46.149812 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41845 (local address 127.0.98.1:53661)
0928 21:15:46.129019 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.129032 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:46.129139 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:46.130408 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:46.130415 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:46.130948 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:46.131429 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.131443 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.132697 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.132705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.133705 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.133714 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.134415 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.134419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.134651 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.142138 (+ 7487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:46.142160 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:46.144251 (+ 2091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:46.146766 (+ 2515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.146769 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.146771 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.147092 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.147529 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.147532 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.147534 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.147625 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.148450 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:46.148456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:46.148808 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 21:15:46.149031 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 21:15:46.149611 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":421,"spinlock_wait_cycles":44032,"thread_start_us":70,"threads_started":1}
I20260928 21:15:46.149845 17354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.128792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53661 (local address 127.0.98.3:41845)
0928 21:15:46.129407 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:46.129424 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:46.129471 (+ 47us) server_negotiation.cc:407] Connection header received
0928 21:15:46.129750 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:46.129755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:46.130068 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:46.130365 (+ 297us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:46.131605 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.132546 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.133829 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.134279 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.134465 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.144445 (+ 9980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:46.144453 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:46.144469 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:46.144507 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:46.146630 (+ 2123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.147232 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.147238 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.147244 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.147421 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.147729 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.147733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.147735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.148263 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:46.148391 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:46.149348 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 21:15:46.149638 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":103,"threads_started":1}
I20260928 21:15:46.149845 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.128884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45079 (local address 127.0.98.2:41497)
0928 21:15:46.129256 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:46.129273 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:46.129478 (+ 205us) server_negotiation.cc:407] Connection header received
0928 21:15:46.129736 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:46.129742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:46.130069 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:46.130305 (+ 236us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:46.131594 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.132546 (+ 952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.133824 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.134273 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.134464 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.144182 (+ 9718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:46.144190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:46.144211 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:46.144246 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:46.146478 (+ 2232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.147165 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.147170 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.147175 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.147327 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.147729 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.147733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.147735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.148263 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:46.148391 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:46.149348 (+ 957us) server_negotiation.cc:299] Negotiation successful
0928 21:15:46.149627 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":79,"threads_started":1}
I20260928 21:15:46.150578 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.128539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41497 (local address 127.0.98.1:45079)
0928 21:15:46.129375 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.129389 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:46.129489 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:46.130663 (+ 1174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:46.130665 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:46.130944 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:46.131428 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.131442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.132691 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.132697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.133703 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.133713 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.134412 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.134415 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.134649 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.142197 (+ 7548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:46.142216 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:46.143917 (+ 1701us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:46.146621 (+ 2704us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.146627 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.146642 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.147038 (+ 396us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.147434 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.147437 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.147442 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.147617 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.148450 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:46.148456 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:46.148808 (+ 352us) client_negotiation.cc:769] Sending connection context
0928 21:15:46.149031 (+ 223us) client_negotiation.cc:240] Negotiation successful
0928 21:15:46.150410 (+ 1379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774,"mutex_wait_us":1827,"spinlock_wait_cycles":12288,"thread_start_us":74,"threads_started":1}
Sep 28 21:15:46 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260928 21:15:46.151929 17297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "5460281e919b48ceb5f1b09c0dd3506e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ed7109ce995249fda574868ff9e62b64" is_pre_election: true
I20260928 21:15:46.151918 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "5460281e919b48ceb5f1b09c0dd3506e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2318442a84d2460fa2441a9b8ff20ff9" is_pre_election: true
I20260928 21:15:46.152176 17162 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5460281e919b48ceb5f1b09c0dd3506e in term 1.
I20260928 21:15:46.152210 17297 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5460281e919b48ceb5f1b09c0dd3506e in term 1.
I20260928 21:15:46.153435 16961 leader_election.cc:302] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2318442a84d2460fa2441a9b8ff20ff9, 5460281e919b48ceb5f1b09c0dd3506e; no voters:
I20260928 21:15:46.153821 17352 raft_consensus.cc:2781] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:15:46.153908 17352 raft_consensus.cc:482] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:15:46.153962 17352 raft_consensus.cc:3037] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 1 FOLLOWER]: Advancing to term 2
Sep 28 21:15:46 dist-test-slave-v7xz krb5kdc[15757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630144, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260928 21:15:46.155565 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.145499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:41845 (local address 127.0.98.2:58507)
0928 21:15:46.145708 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.145729 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:46.145850 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:46.146212 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:46.146220 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:46.146709 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:46.146876 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.146888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.148013 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.148016 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.149047 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.149056 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.149706 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.149710 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.149806 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.150313 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:46.150326 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:46.151841 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:46.153859 (+ 2018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.153867 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.153872 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.154155 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.154496 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.154503 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.154505 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.154594 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.155027 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:46.155033 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:46.155148 (+ 115us) client_negotiation.cc:769] Sending connection context
0928 21:15:46.155267 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:15:46.155412 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117}
I20260928 21:15:46.156061 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.145656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58507 (local address 127.0.98.3:41845)
0928 21:15:46.145926 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:46.145937 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:46.145950 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:15:46.145988 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:46.145991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:46.146034 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:46.146983 (+ 949us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:46.147101 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.147881 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.149192 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.149576 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.151038 (+ 1462us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.151980 (+ 942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:46.151987 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:46.151993 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:46.152025 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:46.153733 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.154289 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.154292 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.154294 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.154390 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.154705 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.154708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.154709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.154904 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:46.155736 (+ 832us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:46.155809 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:15:46.155914 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":86,"threads_started":1}
I20260928 21:15:46.156395 17297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ed7109ce995249fda574868ff9e62b64" is_pre_election: true
I20260928 21:15:46.156540 17297 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2318442a84d2460fa2441a9b8ff20ff9 in term 1.
I20260928 21:15:46.157300 17352 raft_consensus.cc:504] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:46.157308 17097 leader_election.cc:302] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2318442a84d2460fa2441a9b8ff20ff9, ed7109ce995249fda574868ff9e62b64; no voters:
I20260928 21:15:46.157590 17357 raft_consensus.cc:2781] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260928 21:15:46.157683 17357 raft_consensus.cc:482] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:15:46.157737 17357 raft_consensus.cc:3037] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:15:46.157943 17162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "5460281e919b48ceb5f1b09c0dd3506e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "2318442a84d2460fa2441a9b8ff20ff9"
I20260928 21:15:46.157943 17297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "5460281e919b48ceb5f1b09c0dd3506e" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ed7109ce995249fda574868ff9e62b64"
I20260928 21:15:46.158047 17297 raft_consensus.cc:3037] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 1 FOLLOWER]: Advancing to term 2
I20260928 21:15:46.158154 17352 leader_election.cc:288] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [CANDIDATE]: Term 2 election: Requested vote from peers ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845), 2318442a84d2460fa2441a9b8ff20ff9 (127.0.98.2:41497)
I20260928 21:15:46.159847 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.146230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:46749 (local address 127.0.98.2:45711)
0928 21:15:46.147161 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.147176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:15:46.147249 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:15:46.148101 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:15:46.148104 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:15:46.148277 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:15:46.148408 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.148414 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.152084 (+ 3670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.152087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.152951 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:15:46.152961 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.153799 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.153803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:15:46.153891 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.154393 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:15:46.154405 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:15:46.155678 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:15:46.158115 (+ 2437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.158118 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.158120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.158348 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.158673 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:15:46.158678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:15:46.158680 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:15:46.158763 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:15:46.159331 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:15:46.159334 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:15:46.159439 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:15:46.159541 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 21:15:46.159670 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"thread_start_us":76,"threads_started":1}
I20260928 21:15:46.160159 17297 raft_consensus.cc:2454] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5460281e919b48ceb5f1b09c0dd3506e in term 2.
I20260928 21:15:46.160481 16962 leader_election.cc:302] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5460281e919b48ceb5f1b09c0dd3506e, ed7109ce995249fda574868ff9e62b64; no voters:
I20260928 21:15:46.160552 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.146602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45711 (local address 127.0.98.1:46749)
0928 21:15:46.147552 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0928 21:15:46.147559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:15:46.147590 (+ 31us) server_negotiation.cc:407] Connection header received
0928 21:15:46.147629 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:15:46.147634 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:15:46.147824 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:15:46.147955 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:15:46.148534 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.151489 (+ 2955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.153072 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:15:46.153581 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:15:46.153678 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:15:46.155833 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:15:46.155842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:15:46.155853 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:15:46.155885 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:15:46.157972 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.158472 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.158478 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.158479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.158567 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:15:46.158867 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:15:46.158870 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:15:46.158872 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:15:46.159162 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:15:46.159284 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:15:46.160194 (+ 910us) server_negotiation.cc:299] Negotiation successful
0928 21:15:46.160344 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":853,"thread_start_us":85,"threads_started":1}
I20260928 21:15:46.160514 17357 raft_consensus.cc:504] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:46.160784 17162 raft_consensus.cc:2379] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 5460281e919b48ceb5f1b09c0dd3506e in current term 2: Already voted for candidate 2318442a84d2460fa2441a9b8ff20ff9 in this term.
I20260928 21:15:46.160813 17352 raft_consensus.cc:2781] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 2 FOLLOWER]: Leader election won for term 2
I20260928 21:15:46.161062 17357 leader_election.cc:288] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 2 election: Requested vote from peers ed7109ce995249fda574868ff9e62b64 (127.0.98.3:41845), 5460281e919b48ceb5f1b09c0dd3506e (127.0.98.1:46749)
I20260928 21:15:46.161345 17297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "ed7109ce995249fda574868ff9e62b64"
I20260928 21:15:46.161506 17297 raft_consensus.cc:2379] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 2318442a84d2460fa2441a9b8ff20ff9 in current term 2: Already voted for candidate 5460281e919b48ceb5f1b09c0dd3506e in this term.
I20260928 21:15:46.161806 17027 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5460281e919b48ceb5f1b09c0dd3506e" is_pre_election: true
I20260928 21:15:46.162199 17026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e2d714d8cc344d7917e65345ec83c7d" candidate_uuid: "2318442a84d2460fa2441a9b8ff20ff9" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "5460281e919b48ceb5f1b09c0dd3506e"
I20260928 21:15:46.163369 17352 raft_consensus.cc:686] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [term 2 LEADER]: Becoming Leader. State: Replica: 5460281e919b48ceb5f1b09c0dd3506e, State: Running, Role: LEADER
I20260928 21:15:46.163720 17095 leader_election.cc:302] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2318442a84d2460fa2441a9b8ff20ff9; no voters: 5460281e919b48ceb5f1b09c0dd3506e, ed7109ce995249fda574868ff9e62b64
I20260928 21:15:46.163714 17352 consensus_queue.cc:237] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 } } peers { permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 } } peers { permanent_uuid: "5460281e919b48ceb5f1b09c0dd3506e" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 46749 } }
I20260928 21:15:46.164009 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.163755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:46917)
0928 21:15:46.163888 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.163944 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260928 21:15:46.164290 17357 raft_consensus.cc:2726] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260928 21:15:46.607650 17162 raft_consensus.cc:1261] T 4e2d714d8cc344d7917e65345ec83c7d P 2318442a84d2460fa2441a9b8ff20ff9 [term 2 FOLLOWER]: Refusing update from remote peer 5460281e919b48ceb5f1b09c0dd3506e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 21:15:46.608321 17352 consensus_queue.cc:1035] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2318442a84d2460fa2441a9b8ff20ff9" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 21:15:46.610239 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.609928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:41011)
0928 21:15:46.610070 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.610132 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260928 21:15:46.612811 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.612498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:57755)
0928 21:15:46.612674 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.612734 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260928 21:15:46.612883 17297 raft_consensus.cc:1261] T 4e2d714d8cc344d7917e65345ec83c7d P ed7109ce995249fda574868ff9e62b64 [term 2 FOLLOWER]: Refusing update from remote peer 5460281e919b48ceb5f1b09c0dd3506e: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260928 21:15:46.613154 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.612848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:33881)
0928 21:15:46.613026 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.613083 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260928 21:15:46.613428 17352 consensus_queue.cc:1035] T 4e2d714d8cc344d7917e65345ec83c7d P 5460281e919b48ceb5f1b09c0dd3506e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed7109ce995249fda574868ff9e62b64" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 41845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260928 21:15:46.615046 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.614698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:35077)
0928 21:15:46.614894 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.614958 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
I20260928 21:15:46.615938 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:46.615679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:33763)
0928 21:15:46.615802 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:46.615877 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260928 21:15:47.174044 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:47.173676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:43101)
0928 21:15:47.173888 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:47.173940 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":47,"threads_started":1}
I20260928 21:15:47.614221 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:47.613843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:45675)
0928 21:15:47.614070 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:47.614121 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":82,"threads_started":1}
I20260928 21:15:47.616925 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:47.616512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:46667)
0928 21:15:47.616731 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:47.616778 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":80,"threads_started":1}
I20260928 21:15:48.175256 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:48.174831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:34995)
0928 21:15:48.175085 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:48.175147 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":72,"threads_started":1}
I20260928 21:15:48.615470 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:48.615034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:44011)
0928 21:15:48.615303 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:48.615371 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260928 21:15:48.618085 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:48.617611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:37645)
0928 21:15:48.617932 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:48.617993 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":73,"threads_started":1}
I20260928 21:15:49.176460 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:49.176037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:57037)
0928 21:15:49.176303 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:49.176365 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":92,"threads_started":1}
I20260928 21:15:49.616676 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:49.616258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:45507)
0928 21:15:49.616510 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:49.616583 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":86,"threads_started":1}
I20260928 21:15:49.619108 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:49.618749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:60749)
0928 21:15:49.618978 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:49.619026 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
I20260928 21:15:50.178153 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:50.177582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:57457)
0928 21:15:50.177873 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:50.177954 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":108,"threads_started":1}
I20260928 21:15:50.178875 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:50.178624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:55485)
0928 21:15:50.178747 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:50.178794 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 21:15:50.617969 17384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:50.617552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:49917)
0928 21:15:50.617817 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:50.617873 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":98,"threads_started":1}
I20260928 21:15:50.620332 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:50.619932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:51963)
0928 21:15:50.620180 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:50.620245 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":84,"threads_started":1}
I20260928 21:15:51.180141 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:51.179703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:41305)
0928 21:15:51.179979 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:51.180043 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":75,"threads_started":1}
I20260928 21:15:51.619155 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:51.618739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:35657)
0928 21:15:51.618998 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:51.619057 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":79,"threads_started":1}
I20260928 21:15:51.621426 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:51.621061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:51479)
0928 21:15:51.621296 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:51.621346 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260928 21:15:52.181511 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:52.181055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:42791)
0928 21:15:52.181345 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:52.181407 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":95,"threads_started":1}
I20260928 21:15:52.620326 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:52.619900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:40933)
0928 21:15:52.620161 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:52.620233 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":95,"threads_started":1}
I20260928 21:15:52.622453 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:52.622098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:35871)
0928 21:15:52.622323 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:52.622373 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":73,"threads_started":1}
I20260928 21:15:53.182773 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:53.182339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:42965)
0928 21:15:53.182616 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:53.182678 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":94,"threads_started":1}
I20260928 21:15:53.621676 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:53.621220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:46547)
0928 21:15:53.621512 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:53.621579 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":80,"threads_started":1}
I20260928 21:15:53.623608 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:53.623216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:35911)
0928 21:15:53.623460 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:53.623516 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
I20260928 21:15:54.184145 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:54.183619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:36535)
0928 21:15:54.183969 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:54.184040 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":120,"threads_started":1}
I20260928 21:15:54.622828 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:54.622443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:37271)
0928 21:15:54.622677 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:54.622731 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":74,"threads_started":1}
I20260928 21:15:54.624631 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:54.624277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:49799)
0928 21:15:54.624505 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:54.624552 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":91,"threads_started":1}
I20260928 21:15:55.185585 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:55.185200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:33063)
0928 21:15:55.185439 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:55.185492 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
I20260928 21:15:55.624136 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:55.623742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:59141)
0928 21:15:55.623978 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:55.624043 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260928 21:15:55.625816 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:55.625438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:40501)
0928 21:15:55.625667 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:55.625722 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260928 21:15:56.186740 17403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:56.186336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:37539)
0928 21:15:56.186588 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:56.186643 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":74,"threads_started":1}
I20260928 21:15:56.625511 17404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:56.625094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:45761)
0928 21:15:56.625351 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:56.625413 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":80,"threads_started":1}
I20260928 21:15:56.626888 17405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:56.626534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:39727)
0928 21:15:56.626761 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:56.626809 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":91,"threads_started":1}
I20260928 21:15:57.188030 17406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:57.187528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:47311)
0928 21:15:57.187832 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:57.187905 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":108,"threads_started":1}
I20260928 21:15:57.626590 17407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:57.626210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:47647)
0928 21:15:57.626439 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:57.626493 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260928 21:15:57.627912 17408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:57.627565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:49801)
0928 21:15:57.627787 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:57.627837 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":64,"threads_started":1}
I20260928 21:15:58.189465 17410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:58.188996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:43453)
0928 21:15:58.189283 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:58.189344 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260928 21:15:58.628049 17411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:58.627608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:45141)
0928 21:15:58.627886 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:58.627950 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":92,"threads_started":1}
I20260928 21:15:58.629037 17412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:58.628604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:41343)
0928 21:15:58.628879 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:58.628957 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":62,"threads_started":1}
I20260928 21:15:59.190663 17413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:59.190217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.1:49631)
0928 21:15:59.190494 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:59.190554 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":81,"threads_started":1}
I20260928 21:15:59.629387 17414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:59.628931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.2:36103)
0928 21:15:59.629222 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:59.629289 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":77,"threads_started":1}
I20260928 21:15:59.630129 17415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:15:59.629796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41635 (local address 127.0.98.3:45123)
0928 21:15:59.630006 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0928 21:15:59.630055 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":64,"threads_started":1}
I20260928 21:15:59.937443 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 16903
I20260928 21:15:59.943598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17076
I20260928 21:15:59.949411 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17212
2026-09-28T21:15:59Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59275 ms)
[----------] 30 tests from SecurityITest (243822 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:00 dist-test-slave-v7xz krb5kdc[17418](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:00 dist-test-slave-v7xz krb5kdc[17418](info): set up 2 sockets
Sep 28 21:16:00 dist-test-slave-v7xz krb5kdc[17418](info): commencing operation
krb5kdc: starting...
W20260928 21:16:01.004612 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:01.129132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34641
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:01.241379 17431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:01.241695 17431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:01.241777 17431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:01.245900 17431 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:01.245986 17431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:01.246027 17431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:01.246084 17431 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:01.246124 17431 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:01.251574 17431 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:01.252671 17431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:01.253701 17431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:01.262795 17437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:01.262882 17436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:01.262974 17431 server_base.cc:1034] running on GCE node
W20260928 21:16:01.262799 17439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:01.263816 17431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:01.265027 17431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:01.266213 17431 hybrid_clock.cc:648] HybridClock initialized: now 1790630161266178 us; error 49 us; skew 500 ppm
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:01.269161 17431 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:01.270069 17431 webserver.cc:460] Webserver started at http://127.0.98.62:33803/ using document root <none> and password file <none>
I20260928 21:16:01.270642 17431 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:01.270735 17431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:01.270995 17431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:01.273166 17431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "09b8d7c1abc142239608cb0e8301cb61"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "cd5afec783d2b4922f5c5347996e1ad3"
server_key_iv: "ac37379e4ff2c9832bc4bf05520a0440"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.273731 17431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "09b8d7c1abc142239608cb0e8301cb61"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "cd5afec783d2b4922f5c5347996e1ad3"
server_key_iv: "ac37379e4ff2c9832bc4bf05520a0440"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.277500 17431 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260928 21:16:01.280088 17446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.281419 17431 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:01.281574 17431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "09b8d7c1abc142239608cb0e8301cb61"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "cd5afec783d2b4922f5c5347996e1ad3"
server_key_iv: "ac37379e4ff2c9832bc4bf05520a0440"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.281695 17431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:01.295867 17431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:01.299324 17431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:01.299549 17431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:01.306701 17431 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34641
I20260928 21:16:01.306773 17498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34641 every 8 connection(s)
I20260928 21:16:01.307750 17431 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:01.310592 17499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:01.315578 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17431
I20260928 21:16:01.315773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:01.316061 392 external_mini_cluster.cc:1428] Setting key e770d4eda9f89eb80576796db34430f9
I20260928 21:16:01.316331 17499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: Bootstrap starting.
I20260928 21:16:01.318868 17499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:01.319815 17499 log.cc:824] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:01.321950 17499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: No bootstrap required, opened a new log
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:01.326525 17499 raft_consensus.cc:348] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b8d7c1abc142239608cb0e8301cb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34641 } }
I20260928 21:16:01.326789 17499 raft_consensus.cc:374] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:01.326862 17499 raft_consensus.cc:729] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09b8d7c1abc142239608cb0e8301cb61, State: Initialized, Role: FOLLOWER
I20260928 21:16:01.327361 17499 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b8d7c1abc142239608cb0e8301cb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34641 } }
I20260928 21:16:01.327517 17499 raft_consensus.cc:388] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:01.327605 17499 raft_consensus.cc:482] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:01.327725 17499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:01.328632 17499 raft_consensus.cc:504] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b8d7c1abc142239608cb0e8301cb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34641 } }
I20260928 21:16:01.329015 17499 leader_election.cc:302] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09b8d7c1abc142239608cb0e8301cb61; no voters:
I20260928 21:16:01.329363 17499 leader_election.cc:288] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:01.329438 17504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:01.329648 17504 raft_consensus.cc:686] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [term 1 LEADER]: Becoming Leader. State: Replica: 09b8d7c1abc142239608cb0e8301cb61, State: Running, Role: LEADER
I20260928 21:16:01.330005 17504 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b8d7c1abc142239608cb0e8301cb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34641 } }
I20260928 21:16:01.330724 17499 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:01.331727 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:01.317418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58284 (local address 127.0.98.62:34641)
0928 21:16:01.317861 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:01.317876 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:01.317914 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:16:01.318574 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:01.318592 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:01.318940 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:01.319298 (+ 358us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:01.320201 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.321257 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.322397 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.322886 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.323028 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:01.325889 (+ 2861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:01.325895 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:01.325909 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:01.325955 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:01.327916 (+ 1961us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:01.328368 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:01.328374 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:01.328380 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:01.328508 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:01.328884 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:01.328887 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:01.328888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:01.329387 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:01.330224 (+ 837us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:01.330372 (+ 148us) server_negotiation.cc:299] Negotiation successful
0928 21:16:01.330673 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":163,"threads_started":1}
I20260928 21:16:01.331889 17505 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09b8d7c1abc142239608cb0e8301cb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b8d7c1abc142239608cb0e8301cb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34641 } } }
I20260928 21:16:01.332068 17505 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:01.332041 17506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09b8d7c1abc142239608cb0e8301cb61. Latest consensus state: current_term: 1 leader_uuid: "09b8d7c1abc142239608cb0e8301cb61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09b8d7c1abc142239608cb0e8301cb61" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34641 } } }
I20260928 21:16:01.332150 17506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61 [sys.catalog]: This master's current role is: LEADER
W20260928 21:16:01.336685 17519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:01.336776 17519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:01.336906 17513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:01.337620 17513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:01.343945 17513 catalog_manager.cc:1347] Generated new cluster ID: cb3f1242e22f4b77a32be77ed7418c99
I20260928 21:16:01.344048 17513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:01.382828 17513 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:01.384179 17513 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:01.395349 17513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: Generated new TSK 0
I20260928 21:16:01.396304 17513 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:16:01.427611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34641
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:01.543186 17525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:01.543489 17525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:01.543576 17525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:01.547989 17525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:01.548074 17525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:01.548138 17525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:16:01.554216 17525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:01.555372 17525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:01.556531 17525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:01.566558 17530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:01.566546 17533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:01.566586 17531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:01.566622 17525 server_base.cc:1034] running on GCE node
I20260928 21:16:01.567299 17525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:01.567979 17525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:01.569168 17525 hybrid_clock.cc:648] HybridClock initialized: now 1790630161569150 us; error 42 us; skew 500 ppm
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:01.572146 17525 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:16:01.573297 17525 webserver.cc:460] Webserver started at http://127.0.98.1:32859/ using document root <none> and password file <none>
I20260928 21:16:01.573871 17525 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:01.573962 17525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:01.574198 17525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:01.576233 17525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe782ed38c2049588497a738629bce36"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "24d92ce671b24a93466262e2cba57db9"
server_key_iv: "fbf73910a50ff446ef6f312c70e4851e"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.576792 17525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe782ed38c2049588497a738629bce36"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "24d92ce671b24a93466262e2cba57db9"
server_key_iv: "fbf73910a50ff446ef6f312c70e4851e"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.580476 17525 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:01.582943 17540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.584107 17525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:01.584244 17525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "fe782ed38c2049588497a738629bce36"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "24d92ce671b24a93466262e2cba57db9"
server_key_iv: "fbf73910a50ff446ef6f312c70e4851e"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.584374 17525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:01.606288 17525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:01.609699 17525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:01.609907 17525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:01.610508 17525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:01.611340 17525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:01.611413 17525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.611486 17525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:01.611529 17525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.620438 17525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38427
I20260928 21:16:01.620460 17653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38427 every 8 connection(s)
I20260928 21:16:01.621341 17525 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:01.624696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17525
I20260928 21:16:01.624804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:01.625131 392 external_mini_cluster.cc:1428] Setting key 0ef306cc5b9860b96c4848c8e18f5793
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260928 21:16:01.636794 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:01.623118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50659 (local address 127.0.98.62:34641)
0928 21:16:01.623283 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:01.623288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:01.624139 (+ 851us) server_negotiation.cc:407] Connection header received
0928 21:16:01.625555 (+ 1416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:01.625557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:01.625606 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:01.625701 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:01.627129 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.627968 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.629710 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.629956 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.630074 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:01.632979 (+ 2905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:01.632984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:01.632987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:01.633016 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:01.634667 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:01.635191 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:01.635197 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:01.635201 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:01.635302 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:01.635640 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:01.635645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:01.635648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:01.635908 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:01.636017 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:01.636483 (+ 466us) server_negotiation.cc:299] Negotiation successful
0928 21:16:01.636667 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:16:01.637639 17656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:01.623530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34641 (local address 127.0.98.1:50659)
0928 21:16:01.623975 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:01.624018 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:01.625338 (+ 1320us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:01.625841 (+ 503us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:01.625847 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:01.626346 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:01.626972 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:01.626986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.628101 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.628103 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:01.629597 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:01.629607 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.630130 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.630134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:01.630333 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:01.631001 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:01.631030 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:01.632763 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:01.634783 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:01.634789 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:01.634801 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:01.635050 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:01.635404 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:01.635406 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:01.635407 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:01.635532 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:01.636011 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:01.636016 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:01.636260 (+ 244us) client_negotiation.cc:769] Sending connection context
0928 21:16:01.636412 (+ 152us) client_negotiation.cc:240] Negotiation successful
0928 21:16:01.636728 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":124,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:16:01.638950 17654 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34641
I20260928 21:16:01.639209 17654 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:01.639750 17654 heartbeater.cc:511] Master 127.0.98.62:34641 requested a full tablet report, sending...
I20260928 21:16:01.641517 17463 ts_manager.cc:194] Registered new tserver with Master: fe782ed38c2049588497a738629bce36 (127.0.98.1:38427)
I20260928 21:16:01.642668 17463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50659
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:16:01.655189 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34641
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:01.766386 17659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:01.766645 17659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:01.766687 17659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:01.770645 17659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:01.770701 17659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:01.770761 17659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:16:01.775931 17659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:01.776906 17659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:01.777872 17659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:01.787231 17665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:01.787226 17664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:01.787231 17667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:01.787565 17659 server_base.cc:1034] running on GCE node
I20260928 21:16:01.787992 17659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:01.788617 17659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:01.789799 17659 hybrid_clock.cc:648] HybridClock initialized: now 1790630161789771 us; error 71 us; skew 500 ppm
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:01.792459 17659 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:16:01.793433 17659 webserver.cc:460] Webserver started at http://127.0.98.2:42751/ using document root <none> and password file <none>
I20260928 21:16:01.793974 17659 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:01.794083 17659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:01.794310 17659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:01.796235 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "4fc2ef54645a49e88517b8c40e0ea49f"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "c6f8a3ccb3ac30be8f20c8e37f32170f"
server_key_iv: "f1c90b5cbf72c04c2336f5e7dad8fd93"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.796790 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4fc2ef54645a49e88517b8c40e0ea49f"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "c6f8a3ccb3ac30be8f20c8e37f32170f"
server_key_iv: "f1c90b5cbf72c04c2336f5e7dad8fd93"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.800350 17659 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:01.802786 17674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.803990 17659 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:01.804143 17659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "4fc2ef54645a49e88517b8c40e0ea49f"
format_stamp: "Formatted at 2026-09-28 21:16:01 on dist-test-slave-v7xz"
server_key: "c6f8a3ccb3ac30be8f20c8e37f32170f"
server_key_iv: "f1c90b5cbf72c04c2336f5e7dad8fd93"
server_key_version: "encryptionkey@0"
I20260928 21:16:01.804252 17659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:01.826926 17659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:01.829844 17659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:01.830049 17659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:01.830830 17659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:01.831852 17659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:01.831920 17659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.831987 17659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:01.832033 17659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:01.840723 17659 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45413
I20260928 21:16:01.840746 17787 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45413 every 8 connection(s)
I20260928 21:16:01.841717 17659 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
Sep 28 21:16:01 dist-test-slave-v7xz krb5kdc[17418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630161, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:01.851761 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17659
I20260928 21:16:01.851893 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:01.852260 392 external_mini_cluster.cc:1428] Setting key ecd289e699861a94a50ae2c955183d25
I20260928 21:16:01.855561 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:01.843517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59187 (local address 127.0.98.62:34641)
0928 21:16:01.843669 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:01.843674 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:01.844456 (+ 782us) server_negotiation.cc:407] Connection header received
0928 21:16:01.845863 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:01.845866 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:01.845926 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:01.846006 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:01.847188 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.847730 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.848932 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.849160 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.849255 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:01.851572 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:01.851575 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:01.851577 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:01.851600 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:01.853263 (+ 1663us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:01.853868 (+ 605us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:01.853871 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:01.853873 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:01.853963 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:01.854319 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:01.854325 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:01.854328 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:01.854579 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:01.854691 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:01.855196 (+ 505us) server_negotiation.cc:299] Negotiation successful
0928 21:16:01.855331 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:16:01.856470 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:01.843875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34641 (local address 127.0.98.2:59187)
0928 21:16:01.844310 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:01.844353 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:01.845634 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:01.846203 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:01.846210 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:01.846569 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:01.847033 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:01.847047 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.847860 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.847870 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:01.848742 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:01.848752 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:01.849300 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:01.849303 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:01.849444 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:01.849959 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:01.849975 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:01.851393 (+ 1418us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:01.853379 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:01.853385 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:01.853400 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:01.853736 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:01.854058 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:01.854061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:01.854062 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:01.854199 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:01.854711 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:01.854716 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:01.854947 (+ 231us) client_negotiation.cc:769] Sending connection context
0928 21:16:01.855115 (+ 168us) client_negotiation.cc:240] Negotiation successful
0928 21:16:01.855360 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":118,"threads_started":1}
I20260928 21:16:01.857631 17788 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34641
I20260928 21:16:01.857875 17788 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:01.858415 17788 heartbeater.cc:511] Master 127.0.98.62:34641 requested a full tablet report, sending...
I20260928 21:16:01.859516 17463 ts_manager.cc:194] Registered new tserver with Master: 4fc2ef54645a49e88517b8c40e0ea49f (127.0.98.2:45413)
I20260928 21:16:01.860352 17463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59187
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:16:01.879403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34641
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:01.988344 17793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:01.988602 17793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:01.988644 17793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:01.992539 17793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:01.992591 17793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:01.992646 17793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:16:01.997805 17793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34641
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:01.998700 17793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:01.999612 17793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:02.008760 17799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:02.008767 17798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:02.008882 17793 server_base.cc:1034] running on GCE node
W20260928 21:16:02.008767 17801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:02.009415 17793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:02.010062 17793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:02.011226 17793 hybrid_clock.cc:648] HybridClock initialized: now 1790630162011206 us; error 30 us; skew 500 ppm
Sep 28 21:16:02 dist-test-slave-v7xz krb5kdc[17418](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:02.013906 17793 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:16:02.014806 17793 webserver.cc:460] Webserver started at http://127.0.98.3:42177/ using document root <none> and password file <none>
I20260928 21:16:02.015364 17793 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:02.015453 17793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:02.015686 17793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:02.017762 17793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "c34050c5d9414062a01aac9771d11761"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "a74b3c039f4fab9ec5b3a1c10ad7dc92"
server_key_iv: "573252ccc4d0d48ad5ca29bde359385c"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.018318 17793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c34050c5d9414062a01aac9771d11761"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "a74b3c039f4fab9ec5b3a1c10ad7dc92"
server_key_iv: "573252ccc4d0d48ad5ca29bde359385c"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.021879 17793 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:16:02.024144 17808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.025439 17793 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:02.025569 17793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "c34050c5d9414062a01aac9771d11761"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "a74b3c039f4fab9ec5b3a1c10ad7dc92"
server_key_iv: "573252ccc4d0d48ad5ca29bde359385c"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.025679 17793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:02.038959 17793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:02.041795 17793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:02.041991 17793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:02.042555 17793 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:02.043372 17793 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:02.043443 17793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.043514 17793 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:02.043547 17793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.052546 17793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42155
I20260928 21:16:02.052676 17921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42155 every 8 connection(s)
I20260928 21:16:02.053498 17793 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:02.055130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17793
I20260928 21:16:02.055280 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:02.055600 392 external_mini_cluster.cc:1428] Setting key 8d611629b56581b4ef998beb20fdf6b8
Sep 28 21:16:02 dist-test-slave-v7xz krb5kdc[17418](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630162, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:02.067428 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.055439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53121 (local address 127.0.98.62:34641)
0928 21:16:02.055590 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.055594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.056732 (+ 1138us) server_negotiation.cc:407] Connection header received
0928 21:16:02.058329 (+ 1597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.058332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.058384 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.058454 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:02.059586 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.060111 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.061314 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.061675 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.061820 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.064254 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:02.064257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:02.064259 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:02.064283 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:02.065669 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:02.066161 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:02.066164 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:02.066165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:02.066242 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:02.066548 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:02.066550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:02.066551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:02.066714 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:02.066787 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.067218 (+ 431us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.067309 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:16:02.068356 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.055948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34641 (local address 127.0.98.3:53121)
0928 21:16:02.056573 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.056626 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.058128 (+ 1502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.058629 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.058636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.058981 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:02.059444 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.059457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.060231 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.060237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.061197 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.061205 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.061872 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.061877 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.062048 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.062581 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:02.062596 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:02.064095 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:02.065769 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:02.065773 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:02.065785 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:02.066028 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:02.066330 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:02.066332 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:02.066333 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:02.066468 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:02.066796 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:02.066801 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:02.067016 (+ 215us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.067176 (+ 160us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.067413 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":157,"threads_started":1}
I20260928 21:16:02.069564 17922 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34641
I20260928 21:16:02.069789 17922 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:02.070216 17922 heartbeater.cc:511] Master 127.0.98.62:34641 requested a full tablet report, sending...
I20260928 21:16:02.071177 17463 ts_manager.cc:194] Registered new tserver with Master: c34050c5d9414062a01aac9771d11761 (127.0.98.3:42155)
I20260928 21:16:02.071763 17463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53121
I20260928 21:16:02.081324 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:02.090502 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.083243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58294 (local address 127.0.98.62:34641)
0928 21:16:02.083380 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.083384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.083561 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:16:02.083716 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.083719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.083756 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.083835 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:02.084526 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.085081 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.086019 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.086233 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.086338 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.087605 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:02.087608 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:02.087610 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:02.087631 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:02.088998 (+ 1367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:02.089414 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:02.089416 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:02.089418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:02.089488 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:02.089712 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:02.089715 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:02.089716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:02.089916 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:02.090019 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.090200 (+ 181us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.090289 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:16:02.093097 17463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58294:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:02.095974 17463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:02.110288 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.105080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55850 (local address 127.0.98.1:38427)
0928 21:16:02.105917 (+ 837us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.105929 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.105946 (+ 17us) server_negotiation.cc:407] Connection header received
0928 21:16:02.106031 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.106036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.106228 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.106385 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:02.106542 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.107498 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109053 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.109589 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109818 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.109974 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.110026 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.110121 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":84,"threads_started":1}
I20260928 21:16:02.110308 17937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.104700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.0.1:55850)
0928 21:16:02.105638 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.105652 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.105726 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.106360 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.106362 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.106379 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:02.106456 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.106462 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.107809 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.107813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.108873 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.108885 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.109729 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109732 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.109840 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.109875 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.109984 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.110119 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"mutex_wait_us":16,"thread_start_us":84,"threads_started":1}
I20260928 21:16:02.110399 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.104629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37662 (local address 127.0.98.3:42155)
0928 21:16:02.105285 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.105295 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.105318 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:02.105693 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.105701 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.105895 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.106057 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:02.106241 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.107230 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.108718 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.109571 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109680 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.109826 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.110129 (+ 303us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.110229 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":558,"thread_start_us":103,"threads_started":1}
I20260928 21:16:02.110646 17933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.103984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45413 (local address 127.0.0.1:55874)
0928 21:16:02.104351 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.104403 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.104677 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.105418 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.105426 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.105451 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:02.105595 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.105603 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.107013 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.107017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.108487 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.108498 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.109703 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109710 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.110253 (+ 543us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.110276 (+ 23us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.110351 (+ 75us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.110437 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":45,"thread_start_us":86,"threads_started":1}
I20260928 21:16:02.110778 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.104363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42155 (local address 127.0.0.1:37662)
0928 21:16:02.105010 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.105023 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.105508 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.106040 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.106043 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.106059 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:02.106150 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.106155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.107361 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.107366 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.108569 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.108582 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.109890 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109893 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.109979 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.109997 (+ 18us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.110090 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.110166 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":85,"threads_started":1}
I20260928 21:16:02.111166 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.104136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55874 (local address 127.0.98.2:45413)
0928 21:16:02.104848 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.104861 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.104886 (+ 25us) server_negotiation.cc:407] Connection header received
0928 21:16:02.104972 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.104977 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.105209 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.105415 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:02.105718 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.106873 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.108690 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.109568 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.109680 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.109826 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.110906 (+ 1080us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.111023 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":122,"threads_started":1}
I20260928 21:16:02.113639 17856 tablet_service.cc:1467] Processing CreateTablet for tablet 016e3bf09f964c88979be1b4f7fdf1f3 (DEFAULT_TABLE table=test-table [id=383e38f3d4c44557999bf99c41b8f11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:02.113664 17588 tablet_service.cc:1467] Processing CreateTablet for tablet 016e3bf09f964c88979be1b4f7fdf1f3 (DEFAULT_TABLE table=test-table [id=383e38f3d4c44557999bf99c41b8f11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:02.113664 17722 tablet_service.cc:1467] Processing CreateTablet for tablet 016e3bf09f964c88979be1b4f7fdf1f3 (DEFAULT_TABLE table=test-table [id=383e38f3d4c44557999bf99c41b8f11e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:02.114641 17856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016e3bf09f964c88979be1b4f7fdf1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:02.114653 17722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016e3bf09f964c88979be1b4f7fdf1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:02.114653 17588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 016e3bf09f964c88979be1b4f7fdf1f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:02.121485 17939 tablet_bootstrap.cc:492] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Bootstrap starting.
I20260928 21:16:02.122498 17941 tablet_bootstrap.cc:492] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: Bootstrap starting.
I20260928 21:16:02.122686 17940 tablet_bootstrap.cc:492] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Bootstrap starting.
I20260928 21:16:02.123788 17939 tablet_bootstrap.cc:654] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:02.124724 17940 tablet_bootstrap.cc:654] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:02.124724 17941 tablet_bootstrap.cc:654] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:02.124939 17939 log.cc:824] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:02.126066 17940 log.cc:824] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:02.126066 17941 log.cc:824] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:02.127086 17939 tablet_bootstrap.cc:492] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: No bootstrap required, opened a new log
I20260928 21:16:02.127338 17939 ts_tablet_manager.cc:1397] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:02.127992 17941 tablet_bootstrap.cc:492] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: No bootstrap required, opened a new log
I20260928 21:16:02.127992 17940 tablet_bootstrap.cc:492] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: No bootstrap required, opened a new log
I20260928 21:16:02.128180 17941 ts_tablet_manager.cc:1397] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:02.128180 17940 ts_tablet_manager.cc:1397] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:02.130822 17939 raft_consensus.cc:348] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.131093 17939 raft_consensus.cc:374] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:02.131165 17939 raft_consensus.cc:729] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c34050c5d9414062a01aac9771d11761, State: Initialized, Role: FOLLOWER
I20260928 21:16:02.131691 17939 consensus_queue.cc:260] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.132419 17922 heartbeater.cc:503] Master 127.0.98.62:34641 was elected leader, sending a full tablet report...
I20260928 21:16:02.132179 17940 raft_consensus.cc:348] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.132526 17940 raft_consensus.cc:374] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:02.132584 17940 raft_consensus.cc:729] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fc2ef54645a49e88517b8c40e0ea49f, State: Initialized, Role: FOLLOWER
I20260928 21:16:02.132932 17939 ts_tablet_manager.cc:1428] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Time spent starting tablet: real 0.006s user 0.002s sys 0.005s
I20260928 21:16:02.133034 17940 consensus_queue.cc:260] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.132177 17941 raft_consensus.cc:348] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.133566 17941 raft_consensus.cc:374] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:02.133657 17941 raft_consensus.cc:729] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe782ed38c2049588497a738629bce36, State: Initialized, Role: FOLLOWER
I20260928 21:16:02.134100 17788 heartbeater.cc:503] Master 127.0.98.62:34641 was elected leader, sending a full tablet report...
I20260928 21:16:02.134100 17941 consensus_queue.cc:260] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.134770 17654 heartbeater.cc:503] Master 127.0.98.62:34641 was elected leader, sending a full tablet report...
I20260928 21:16:02.135128 17941 ts_tablet_manager.cc:1428] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260928 21:16:02.135802 17940 ts_tablet_manager.cc:1428] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
W20260928 21:16:02.304507 17923 tablet.cc:2369] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:02.342890 17789 tablet.cc:2369] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:02.372763 17655 tablet.cc:2369] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:02.383832 17947 raft_consensus.cc:482] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:02.384042 17947 raft_consensus.cc:504] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.385425 17947 leader_election.cc:288] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fc2ef54645a49e88517b8c40e0ea49f (127.0.98.2:45413), c34050c5d9414062a01aac9771d11761 (127.0.98.3:42155)
I20260928 21:16:02.389278 17948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.385636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45413 (local address 127.0.98.1:34929)
0928 21:16:02.385933 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.385948 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.386089 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.386397 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.386400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.386417 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:02.386547 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.386552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.387253 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.387259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.388160 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.388170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.388791 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.388794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.388920 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.388948 (+ 28us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.389044 (+ 96us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.389117 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260928 21:16:02.389261 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.385729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34929 (local address 127.0.98.2:45413)
0928 21:16:02.385850 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.385856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.386041 (+ 185us) server_negotiation.cc:407] Connection header received
0928 21:16:02.386238 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.386242 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.386296 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.386373 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:02.386653 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.387149 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.388276 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.388671 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.388787 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.388870 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.389059 (+ 189us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.389130 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260928 21:16:02.389446 17936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.385977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34633 (local address 127.0.98.3:42155)
0928 21:16:02.386106 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.386111 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.386211 (+ 100us) server_negotiation.cc:407] Connection header received
0928 21:16:02.386362 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.386369 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.386411 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.386491 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:02.386750 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.387233 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.388371 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.388778 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.389022 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.389084 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.389260 (+ 176us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.389326 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260928 21:16:02.389453 17949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.385713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42155 (local address 127.0.98.1:34633)
0928 21:16:02.386106 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.386121 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.386259 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.386521 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.386524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.386546 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:02.386646 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.386651 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.387345 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.387350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.388249 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.388258 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.389006 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.389011 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.389110 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.389135 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.389240 (+ 105us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.389309 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":120,"threads_started":1}
I20260928 21:16:02.390256 17742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016e3bf09f964c88979be1b4f7fdf1f3" candidate_uuid: "fe782ed38c2049588497a738629bce36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" is_pre_election: true
I20260928 21:16:02.390254 17876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016e3bf09f964c88979be1b4f7fdf1f3" candidate_uuid: "fe782ed38c2049588497a738629bce36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34050c5d9414062a01aac9771d11761" is_pre_election: true
I20260928 21:16:02.390568 17742 raft_consensus.cc:2454] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe782ed38c2049588497a738629bce36 in term 0.
I20260928 21:16:02.390579 17876 raft_consensus.cc:2454] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe782ed38c2049588497a738629bce36 in term 0.
I20260928 21:16:02.391044 17544 leader_election.cc:302] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc2ef54645a49e88517b8c40e0ea49f, fe782ed38c2049588497a738629bce36; no voters:
I20260928 21:16:02.391456 17947 raft_consensus.cc:2781] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:02.391556 17947 raft_consensus.cc:482] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:02.391615 17947 raft_consensus.cc:3037] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:02.392688 17947 raft_consensus.cc:504] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.393340 17947 leader_election.cc:288] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [CANDIDATE]: Term 1 election: Requested vote from peers 4fc2ef54645a49e88517b8c40e0ea49f (127.0.98.2:45413), c34050c5d9414062a01aac9771d11761 (127.0.98.3:42155)
I20260928 21:16:02.393436 17742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016e3bf09f964c88979be1b4f7fdf1f3" candidate_uuid: "fe782ed38c2049588497a738629bce36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fc2ef54645a49e88517b8c40e0ea49f"
I20260928 21:16:02.393532 17876 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "016e3bf09f964c88979be1b4f7fdf1f3" candidate_uuid: "fe782ed38c2049588497a738629bce36" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c34050c5d9414062a01aac9771d11761"
I20260928 21:16:02.393565 17742 raft_consensus.cc:3037] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:02.393631 17876 raft_consensus.cc:3037] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:02.394618 17742 raft_consensus.cc:2454] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe782ed38c2049588497a738629bce36 in term 1.
I20260928 21:16:02.394713 17876 raft_consensus.cc:2454] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe782ed38c2049588497a738629bce36 in term 1.
I20260928 21:16:02.394951 17544 leader_election.cc:302] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fc2ef54645a49e88517b8c40e0ea49f, fe782ed38c2049588497a738629bce36; no voters:
I20260928 21:16:02.395215 17947 raft_consensus.cc:2781] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:02.395462 17947 raft_consensus.cc:686] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [term 1 LEADER]: Becoming Leader. State: Replica: fe782ed38c2049588497a738629bce36, State: Running, Role: LEADER
I20260928 21:16:02.396072 17947 consensus_queue.cc:237] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } }
I20260928 21:16:02.399436 17462 catalog_manager.cc:5495] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe782ed38c2049588497a738629bce36 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fe782ed38c2049588497a738629bce36" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe782ed38c2049588497a738629bce36" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:02.474570 17938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.471046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.0.98.1:38427)
0928 21:16:02.471198 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.471201 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.471215 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:02.471325 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.471327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.471382 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.471489 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:02.471807 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.472363 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.473250 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.473455 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.473539 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.473786 (+ 247us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:02.474186 (+ 400us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:02.474274 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.474327 (+ 53us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.474372 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:16:02.480334 17876 raft_consensus.cc:1261] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 1 FOLLOWER]: Refusing update from remote peer fe782ed38c2049588497a738629bce36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:02.480334 17742 raft_consensus.cc:1261] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 1 FOLLOWER]: Refusing update from remote peer fe782ed38c2049588497a738629bce36: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:02.481083 17950 consensus_queue.cc:1035] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c34050c5d9414062a01aac9771d11761" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:02.481287 17947 consensus_queue.cc:1035] T 016e3bf09f964c88979be1b4f7fdf1f3 P fe782ed38c2049588497a738629bce36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fc2ef54645a49e88517b8c40e0ea49f" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:02.488365 17953 mvcc.cc:205] Tried to move back new op lower bound from 7334421145512546304 to 7334421145177174016. Current Snapshot: MvccSnapshot[applied={T|T < 7334421145512546304}]
I20260928 21:16:02.489512 17954 mvcc.cc:205] Tried to move back new op lower bound from 7334421145512546304 to 7334421145177174016. Current Snapshot: MvccSnapshot[applied={T|T < 7334421145512546304}]
I20260928 21:16:02.497484 17461 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:02.497684 17461 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58294:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:02.500105 17461 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 09b8d7c1abc142239608cb0e8301cb61: Sending DeleteTablet for 3 replicas of tablet 016e3bf09f964c88979be1b4f7fdf1f3
I20260928 21:16:02.500967 17722 tablet_service.cc:1514] Processing DeleteTablet for tablet 016e3bf09f964c88979be1b4f7fdf1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55874
I20260928 21:16:02.501039 17588 tablet_service.cc:1514] Processing DeleteTablet for tablet 016e3bf09f964c88979be1b4f7fdf1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55850
I20260928 21:16:02.501557 17964 tablet_replica.cc:330] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: stopping tablet replica
I20260928 21:16:02.501542 17856 tablet_service.cc:1514] Processing DeleteTablet for tablet 016e3bf09f964c88979be1b4f7fdf1f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:02 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37662
I20260928 21:16:02.501909 17964 raft_consensus.cc:2229] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:02.502043 17965 tablet_replica.cc:330] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: stopping tablet replica
I20260928 21:16:02.502274 17964 raft_consensus.cc:2258] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:02.502434 17965 raft_consensus.cc:2229] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:02.502687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17525
I20260928 21:16:02.502770 17965 raft_consensus.cc:2258] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:02.503885 17964 ts_tablet_manager.cc:1905] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:02.504129 17965 ts_tablet_manager.cc:1905] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:02.506618 17964 ts_tablet_manager.cc:1918] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:02.506618 17965 ts_tablet_manager.cc:1918] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:02.506724 17964 log.cc:1196] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/016e3bf09f964c88979be1b4f7fdf1f3
I20260928 21:16:02.506724 17965 log.cc:1196] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/016e3bf09f964c88979be1b4f7fdf1f3
I20260928 21:16:02.507058 17964 ts_tablet_manager.cc:1939] T 016e3bf09f964c88979be1b4f7fdf1f3 P 4fc2ef54645a49e88517b8c40e0ea49f: Deleting consensus metadata
I20260928 21:16:02.507076 17965 ts_tablet_manager.cc:1939] T 016e3bf09f964c88979be1b4f7fdf1f3 P c34050c5d9414062a01aac9771d11761: Deleting consensus metadata
I20260928 21:16:02.508277 17450 catalog_manager.cc:4841] TS 4fc2ef54645a49e88517b8c40e0ea49f (127.0.98.2:45413): tablet 016e3bf09f964c88979be1b4f7fdf1f3 (table test-table [id=383e38f3d4c44557999bf99c41b8f11e]) successfully deleted
I20260928 21:16:02.508520 17447 catalog_manager.cc:4841] TS c34050c5d9414062a01aac9771d11761 (127.0.98.3:42155): tablet 016e3bf09f964c88979be1b4f7fdf1f3 (table test-table [id=383e38f3d4c44557999bf99c41b8f11e]) successfully deleted
W20260928 21:16:02.512630 17450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38427 (error 108)
I20260928 21:16:02.512764 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17659
I20260928 21:16:02.515230 17935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.514959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38427 (local address 127.0.0.1:55866)
0928 21:16:02.515100 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.515158 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 21:16:02.515431 17450 catalog_manager.cc:4584] TS fe782ed38c2049588497a738629bce36 (127.0.98.1:38427): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 016e3bf09f964c88979be1b4f7fdf1f3: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38427: connect: Connection refused (error 111)
I20260928 21:16:02.518566 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17793
I20260928 21:16:02.523710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17431
2026-09-28T21:16:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2573 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-28T21:16:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:02Z Disabled control of system clock
I20260928 21:16:02.553606 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34651
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34651
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:16:02.666780 17970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:02.667047 17970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:02.667081 17970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:02.671057 17970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:02.671121 17970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:02.671136 17970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:02.671154 17970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:02.671169 17970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:02.675922 17970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34651
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:02.677007 17970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:02.678334 17970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:02.687119 17978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:02.687203 17970 server_base.cc:1034] running on GCE node
W20260928 21:16:02.687098 17976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:02.687098 17975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:02.687978 17970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:02.689194 17970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:02.690353 17970 hybrid_clock.cc:648] HybridClock initialized: now 1790630162690340 us; error 30 us; skew 500 ppm
I20260928 21:16:02.692265 17970 webserver.cc:460] Webserver started at http://127.0.98.62:40817/ using document root <none> and password file <none>
I20260928 21:16:02.692927 17970 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:02.693002 17970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:02.693236 17970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:02.695293 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "88c7da9f5c7b4b1182884eac10996675"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "d94d65f21b5b8a1ebf71e5ec80097f16"
server_key_iv: "9b7935d96b3da013bec9e57e793eaf0a"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.695874 17970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "88c7da9f5c7b4b1182884eac10996675"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "d94d65f21b5b8a1ebf71e5ec80097f16"
server_key_iv: "9b7935d96b3da013bec9e57e793eaf0a"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.699684 17970 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:16:02.702231 17984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.703492 17970 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:02.703634 17970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "88c7da9f5c7b4b1182884eac10996675"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "d94d65f21b5b8a1ebf71e5ec80097f16"
server_key_iv: "9b7935d96b3da013bec9e57e793eaf0a"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.703748 17970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:02.722335 17970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:02.723182 17970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:02.723402 17970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:02.730578 17970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34651
I20260928 21:16:02.730588 18036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34651 every 8 connection(s)
I20260928 21:16:02.731534 17970 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:02.733402 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 17970
I20260928 21:16:02.733572 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:02.733857 392 external_mini_cluster.cc:1428] Setting key f3674fd83171a034955bcfc6aa23553c
I20260928 21:16:02.735170 18037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:02.741863 18037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675: Bootstrap starting.
I20260928 21:16:02.743146 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.735186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50910 (local address 127.0.98.62:34651)
0928 21:16:02.735825 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.735840 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.735875 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:16:02.736513 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.736529 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.737011 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.737307 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:02.737698 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.738661 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.739875 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.740346 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.740463 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.741398 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:02.741403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:02.741415 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:02.741826 (+ 411us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:02.741880 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:02.741899 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:02.742030 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.742181 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.742440 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":138,"threads_started":1}
I20260928 21:16:02.744320 18037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:02.745250 18037 log.cc:824] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:02.747218 18037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675: No bootstrap required, opened a new log
I20260928 21:16:02.750756 18037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c7da9f5c7b4b1182884eac10996675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34651 } }
I20260928 21:16:02.751075 18037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:02.751194 18037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c7da9f5c7b4b1182884eac10996675, State: Initialized, Role: FOLLOWER
I20260928 21:16:02.751710 18037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c7da9f5c7b4b1182884eac10996675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34651 } }
I20260928 21:16:02.751897 18037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:02.751987 18037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:02.752085 18037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:02.753067 18037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c7da9f5c7b4b1182884eac10996675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34651 } }
I20260928 21:16:02.753407 18037 leader_election.cc:302] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88c7da9f5c7b4b1182884eac10996675; no voters:
I20260928 21:16:02.753717 18037 leader_election.cc:288] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:02.753826 18042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:02.754082 18042 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [term 1 LEADER]: Becoming Leader. State: Replica: 88c7da9f5c7b4b1182884eac10996675, State: Running, Role: LEADER
I20260928 21:16:02.754360 18042 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c7da9f5c7b4b1182884eac10996675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34651 } }
I20260928 21:16:02.754987 18037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:02.755867 18043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88c7da9f5c7b4b1182884eac10996675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c7da9f5c7b4b1182884eac10996675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34651 } } }
I20260928 21:16:02.755982 18043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:02.756306 18044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88c7da9f5c7b4b1182884eac10996675. Latest consensus state: current_term: 1 leader_uuid: "88c7da9f5c7b4b1182884eac10996675" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88c7da9f5c7b4b1182884eac10996675" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34651 } } }
I20260928 21:16:02.756399 18044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:02.757437 18051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:02.760406 18051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:02.766428 18051 catalog_manager.cc:1347] Generated new cluster ID: 1019b36a7def46d291e45f8e6e7f6411
I20260928 21:16:02.766498 18051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:02.778550 18051 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:02.779757 18051 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:02.788619 18051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675: Generated new TSK 0
I20260928 21:16:02.789347 18051 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:02.795770 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34651
--builtin_ntp_servers=127.0.98.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:16:02.904222 18061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:02.904484 18061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:02.904522 18061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:02.908614 18061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:02.908664 18061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:02.908763 18061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:16:02.913478 18061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:02.914525 18061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:02.915865 18061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:02.925995 18069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:02.925987 18067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:02.925987 18066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:02.926194 18061 server_base.cc:1034] running on GCE node
I20260928 21:16:02.926632 18061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:02.927273 18061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:02.928458 18061 hybrid_clock.cc:648] HybridClock initialized: now 1790630162928435 us; error 36 us; skew 500 ppm
I20260928 21:16:02.930379 18061 webserver.cc:460] Webserver started at http://127.0.98.1:37019/ using document root <none> and password file <none>
I20260928 21:16:02.930964 18061 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:02.931057 18061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:02.931288 18061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:02.933421 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "219fd7adecb042c5991542e1a16f88e3"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "f9bf25ea5a62de8e1ac08f6e15f0bf12"
server_key_iv: "b68fd698e14574a895cd2bc2c8915b06"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.933979 18061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "219fd7adecb042c5991542e1a16f88e3"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "f9bf25ea5a62de8e1ac08f6e15f0bf12"
server_key_iv: "b68fd698e14574a895cd2bc2c8915b06"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.937583 18061 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:02.939925 18075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.941193 18061 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:02.941323 18061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "219fd7adecb042c5991542e1a16f88e3"
format_stamp: "Formatted at 2026-09-28 21:16:02 on dist-test-slave-v7xz"
server_key: "f9bf25ea5a62de8e1ac08f6e15f0bf12"
server_key_iv: "b68fd698e14574a895cd2bc2c8915b06"
server_key_version: "encryptionkey@0"
I20260928 21:16:02.941447 18061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:02.953794 18061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:02.954479 18061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:02.954681 18061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:02.955258 18061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:02.956293 18061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:02.956359 18061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.956429 18061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:02.956465 18061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:02.965055 18061 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45625
I20260928 21:16:02.965075 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45625 every 8 connection(s)
I20260928 21:16:02.966054 18061 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:02.972034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18061
I20260928 21:16:02.972215 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:02.972594 392 external_mini_cluster.cc:1428] Setting key d3950fc07048f4a430eaa5443fda9538
I20260928 21:16:02.974809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34651
--builtin_ntp_servers=127.0.98.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:16:02.976261 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.967851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52595 (local address 127.0.98.62:34651)
0928 21:16:02.967989 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:02.967993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:02.969084 (+ 1091us) server_negotiation.cc:407] Connection header received
0928 21:16:02.970502 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:02.970505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:02.970575 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:02.970674 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:02.971588 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.972135 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.973767 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.974117 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.974210 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.975682 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:02.975685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:02.975687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:02.975699 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:02.975717 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:02.975725 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:02.975786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:02.975979 (+ 193us) server_negotiation.cc:299] Negotiation successful
0928 21:16:02.976071 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:16:02.977267 18191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:02.968224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34651 (local address 127.0.98.1:52595)
0928 21:16:02.968941 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:02.968982 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:02.970267 (+ 1285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:02.970858 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:02.970865 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:02.970956 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:02.971442 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.971454 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.972350 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.972354 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.973639 (+ 1285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:02.973659 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:02.974293 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:02.974298 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:02.974508 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:02.975253 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:02.975258 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:02.975321 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:02.975414 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:02.975814 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:02.975826 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:02.975835 (+ 9us) client_negotiation.cc:769] Sending connection context
0928 21:16:02.975978 (+ 143us) client_negotiation.cc:240] Negotiation successful
0928 21:16:02.976347 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":112,"threads_started":1}
I20260928 21:16:02.978638 18189 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34651
I20260928 21:16:02.978899 18189 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:02.979434 18189 heartbeater.cc:511] Master 127.0.98.62:34651 requested a full tablet report, sending...
I20260928 21:16:02.981021 18001 ts_manager.cc:194] Registered new tserver with Master: 219fd7adecb042c5991542e1a16f88e3 (127.0.98.1:45625)
I20260928 21:16:02.982185 18001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:52595
W20260928 21:16:03.090579 18192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:03.090860 18192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:03.090901 18192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:03.095098 18192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:03.095155 18192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:03.095211 18192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:16:03.100169 18192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:03.101336 18192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:03.102835 18192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:03.114768 18198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:03.115361 18197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:03.115597 18200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:03.116336 18192 server_base.cc:1034] running on GCE node
I20260928 21:16:03.116811 18192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:03.117617 18192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:03.118834 18192 hybrid_clock.cc:648] HybridClock initialized: now 1790630163118807 us; error 43 us; skew 500 ppm
I20260928 21:16:03.120973 18192 webserver.cc:460] Webserver started at http://127.0.98.2:43205/ using document root <none> and password file <none>
I20260928 21:16:03.121713 18192 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:03.121805 18192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:03.122068 18192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:03.124168 18192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "514bb6c484bd4dd7b61789115254452a"
format_stamp: "Formatted at 2026-09-28 21:16:03 on dist-test-slave-v7xz"
server_key: "be578156a3eca5cf08be91b487e73379"
server_key_iv: "ec7f0afd6a38fee88f02df716946a7e9"
server_key_version: "encryptionkey@0"
I20260928 21:16:03.124758 18192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "514bb6c484bd4dd7b61789115254452a"
format_stamp: "Formatted at 2026-09-28 21:16:03 on dist-test-slave-v7xz"
server_key: "be578156a3eca5cf08be91b487e73379"
server_key_iv: "ec7f0afd6a38fee88f02df716946a7e9"
server_key_version: "encryptionkey@0"
I20260928 21:16:03.128592 18192 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260928 21:16:03.131150 18206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:03.132375 18192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:03.132513 18192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "514bb6c484bd4dd7b61789115254452a"
format_stamp: "Formatted at 2026-09-28 21:16:03 on dist-test-slave-v7xz"
server_key: "be578156a3eca5cf08be91b487e73379"
server_key_iv: "ec7f0afd6a38fee88f02df716946a7e9"
server_key_version: "encryptionkey@0"
I20260928 21:16:03.132633 18192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:03.148778 18192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:03.149621 18192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:03.149845 18192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:03.150493 18192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:03.151558 18192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:03.151639 18192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:03.151773 18192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:03.151816 18192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:03.162074 18192 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45337
I20260928 21:16:03.162094 18319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45337 every 8 connection(s)
I20260928 21:16:03.162988 18192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:03.163787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18192
I20260928 21:16:03.163933 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:03.164225 392 external_mini_cluster.cc:1428] Setting key 947dab7c89c68fe52294bb9eadcd1953
I20260928 21:16:03.166447 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34651
--builtin_ntp_servers=127.0.98.20:38931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:16:03.172785 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.164961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40457 (local address 127.0.98.62:34651)
0928 21:16:03.165107 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.165113 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.166299 (+ 1186us) server_negotiation.cc:407] Connection header received
0928 21:16:03.167883 (+ 1584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.167886 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.167932 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.168003 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:03.169152 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.169660 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.170814 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.171037 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.171149 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.172292 (+ 1143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:03.172301 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:03.172303 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:03.172316 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:03.172333 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:03.172341 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:03.172428 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.172580 (+ 152us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.172644 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:16:03.173666 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.165436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34651 (local address 127.0.98.2:40457)
0928 21:16:03.166130 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.166191 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.167639 (+ 1448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.168195 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.168206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.168324 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:03.169029 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.169045 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.169794 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.169803 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.170686 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.170696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.171189 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.171193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.171356 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.172023 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:03.172024 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:03.172065 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:03.172126 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:03.172424 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:03.172434 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:03.172442 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.172580 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.172881 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":139,"threads_started":1}
I20260928 21:16:03.174964 18320 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34651
I20260928 21:16:03.175200 18320 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:03.175712 18320 heartbeater.cc:511] Master 127.0.98.62:34651 requested a full tablet report, sending...
I20260928 21:16:03.176759 18001 ts_manager.cc:194] Registered new tserver with Master: 514bb6c484bd4dd7b61789115254452a (127.0.98.2:45337)
I20260928 21:16:03.177521 18001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:40457
W20260928 21:16:03.280336 18323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:03.280640 18323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:03.280720 18323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:03.285248 18323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:03.285336 18323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:03.285434 18323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:16:03.290645 18323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34651
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:03.291755 18323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:03.293401 18323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:03.302556 18329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:03.302568 18331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:03.302568 18328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:03.302836 18323 server_base.cc:1034] running on GCE node
I20260928 21:16:03.303351 18323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:03.303912 18323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:03.305090 18323 hybrid_clock.cc:648] HybridClock initialized: now 1790630163305057 us; error 48 us; skew 500 ppm
I20260928 21:16:03.306777 18323 webserver.cc:460] Webserver started at http://127.0.98.3:44677/ using document root <none> and password file <none>
I20260928 21:16:03.307425 18323 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:03.307516 18323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:03.307730 18323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:03.309881 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "9c17c02162144c4d8793a10ad3fb0460"
format_stamp: "Formatted at 2026-09-28 21:16:03 on dist-test-slave-v7xz"
server_key: "c6064f1604fd2152d2bd5a6692a0e34a"
server_key_iv: "d93bfad3b9f0865726ffd877efc8e1cc"
server_key_version: "encryptionkey@0"
I20260928 21:16:03.310447 18323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "9c17c02162144c4d8793a10ad3fb0460"
format_stamp: "Formatted at 2026-09-28 21:16:03 on dist-test-slave-v7xz"
server_key: "c6064f1604fd2152d2bd5a6692a0e34a"
server_key_iv: "d93bfad3b9f0865726ffd877efc8e1cc"
server_key_version: "encryptionkey@0"
I20260928 21:16:03.314057 18323 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:03.316371 18337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:03.317588 18323 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:03.317727 18323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "9c17c02162144c4d8793a10ad3fb0460"
format_stamp: "Formatted at 2026-09-28 21:16:03 on dist-test-slave-v7xz"
server_key: "c6064f1604fd2152d2bd5a6692a0e34a"
server_key_iv: "d93bfad3b9f0865726ffd877efc8e1cc"
server_key_version: "encryptionkey@0"
I20260928 21:16:03.317838 18323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:03.329767 18323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:03.330391 18323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:03.330577 18323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:03.331183 18323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:03.332134 18323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:03.332223 18323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:03.332295 18323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:03.332347 18323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:03.341042 18323 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37897
I20260928 21:16:03.341061 18450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37897 every 8 connection(s)
I20260928 21:16:03.341972 18323 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:03.343092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18323
I20260928 21:16:03.343209 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:03.343497 392 external_mini_cluster.cc:1428] Setting key ec2c653c2ed70b78f897704cb88ac960
I20260928 21:16:03.351223 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.343849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39471 (local address 127.0.98.62:34651)
0928 21:16:03.344034 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.344040 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.345338 (+ 1298us) server_negotiation.cc:407] Connection header received
0928 21:16:03.346908 (+ 1570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.346911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.346955 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.347027 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:03.347911 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.348416 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.349579 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.349790 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.349869 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.350833 (+ 964us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:03.350836 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:03.350838 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:03.350849 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:03.350868 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:03.350875 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:03.350932 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.351063 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.351112 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260928 21:16:03.352053 18453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.344394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34651 (local address 127.0.98.3:39471)
0928 21:16:03.345180 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.345234 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.346693 (+ 1459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.347189 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.347196 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.347281 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:03.347794 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.347805 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.348517 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.348520 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.349474 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.349483 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.349920 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.349923 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.350058 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.350598 (+ 540us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:03.350599 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:03.350638 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:03.350699 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:03.350947 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:03.350953 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:03.350957 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.351049 (+ 92us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.351262 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260928 21:16:03.353168 18451 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34651
I20260928 21:16:03.353411 18451 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:03.353927 18451 heartbeater.cc:511] Master 127.0.98.62:34651 requested a full tablet report, sending...
I20260928 21:16:03.354905 18001 ts_manager.cc:194] Registered new tserver with Master: 9c17c02162144c4d8793a10ad3fb0460 (127.0.98.3:37897)
I20260928 21:16:03.355511 18001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:39471
I20260928 21:16:03.357527 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:03.362746 18040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.359335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50914 (local address 127.0.98.62:34651)
0928 21:16:03.359478 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.359484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.359603 (+ 119us) server_negotiation.cc:407] Connection header received
0928 21:16:03.359692 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.359694 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.359731 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.359790 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:03.360005 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.360484 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.361317 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.361530 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.361608 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.362365 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:03.362368 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:03.362370 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:03.362380 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:03.362394 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:03.362401 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:03.362454 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.362536 (+ 82us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.362576 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:16:03.365578 18001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50914:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:03.368618 18001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:03.382774 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.377336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49926 (local address 127.0.98.3:37897)
0928 21:16:03.377753 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.377765 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.377794 (+ 29us) server_negotiation.cc:407] Connection header received
0928 21:16:03.377874 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.377880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.378101 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.378637 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:03.378779 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.379698 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.381208 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.381998 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.382104 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.382293 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.382512 (+ 219us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.382613 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":142,"threads_started":1}
I20260928 21:16:03.382767 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.377171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37897 (local address 127.0.0.1:49926)
0928 21:16:03.377558 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.377616 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.377738 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.378288 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.378294 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.378319 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:03.378481 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.378489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.379818 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.379821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.381087 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.381098 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.382121 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.382124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.382286 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.382327 (+ 41us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.382473 (+ 146us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.382597 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":24,"thread_start_us":92,"threads_started":1}
I20260928 21:16:03.382917 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.377530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45337 (local address 127.0.0.1:42976)
0928 21:16:03.378165 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.378179 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.378266 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.378887 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.378890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.378915 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:03.379014 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.379021 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.380327 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.380331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.381485 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.381493 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.382541 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.382544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.382623 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.382642 (+ 19us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.382689 (+ 47us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.382738 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":90,"threads_started":1}
I20260928 21:16:03.383493 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.377982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42976 (local address 127.0.98.2:45337)
0928 21:16:03.378511 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.378525 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.378549 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:16:03.378622 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.378626 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.378753 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.378904 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:03.379112 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.380206 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.381617 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.382417 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.382977 (+ 560us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.383164 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.383219 (+ 55us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.383318 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":137,"threads_started":1}
I20260928 21:16:03.384253 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.377912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45625 (local address 127.0.0.1:54898)
0928 21:16:03.378371 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.378383 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.378457 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.379712 (+ 1255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.379715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.379738 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:03.379837 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.379843 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.381078 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.381084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.382264 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.382274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.383331 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.383334 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.383425 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.383450 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.383500 (+ 50us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.384129 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":29,"thread_start_us":89,"threads_started":1}
I20260928 21:16:03.384992 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.378301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54898 (local address 127.0.98.1:45625)
0928 21:16:03.379297 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.379308 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.379332 (+ 24us) server_negotiation.cc:407] Connection header received
0928 21:16:03.379403 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.379408 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.379570 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.379710 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:03.379969 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.380956 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.382371 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.383194 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.384453 (+ 1259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.384639 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.384693 (+ 54us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.384783 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":893,"thread_start_us":106,"threads_started":1}
I20260928 21:16:03.386198 18385 tablet_service.cc:1467] Processing CreateTablet for tablet 4a04fda056524d478809888a3fb11583 (DEFAULT_TABLE table=test-table [id=11cb4f36b4494656a730b7deaaae17e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:03.387136 18385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a04fda056524d478809888a3fb11583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:03.390851 18254 tablet_service.cc:1467] Processing CreateTablet for tablet 4a04fda056524d478809888a3fb11583 (DEFAULT_TABLE table=test-table [id=11cb4f36b4494656a730b7deaaae17e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:03.391947 18254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a04fda056524d478809888a3fb11583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:03.394249 18123 tablet_service.cc:1467] Processing CreateTablet for tablet 4a04fda056524d478809888a3fb11583 (DEFAULT_TABLE table=test-table [id=11cb4f36b4494656a730b7deaaae17e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:03.395313 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a04fda056524d478809888a3fb11583. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:03.397058 18468 tablet_bootstrap.cc:492] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Bootstrap starting.
I20260928 21:16:03.398994 18469 tablet_bootstrap.cc:492] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Bootstrap starting.
I20260928 21:16:03.400457 18468 tablet_bootstrap.cc:654] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:03.401199 18469 tablet_bootstrap.cc:654] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:03.401414 18468 log.cc:824] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:03.401994 18469 log.cc:824] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:03.402699 18472 tablet_bootstrap.cc:492] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: Bootstrap starting.
I20260928 21:16:03.403625 18468 tablet_bootstrap.cc:492] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: No bootstrap required, opened a new log
I20260928 21:16:03.403856 18468 ts_tablet_manager.cc:1397] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:16:03.404036 18469 tablet_bootstrap.cc:492] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: No bootstrap required, opened a new log
I20260928 21:16:03.404241 18469 ts_tablet_manager.cc:1397] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:16:03.405225 18472 tablet_bootstrap.cc:654] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:03.406108 18472 log.cc:824] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:03.408062 18469 raft_consensus.cc:348] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.408327 18469 raft_consensus.cc:374] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:03.408370 18472 tablet_bootstrap.cc:492] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: No bootstrap required, opened a new log
I20260928 21:16:03.408389 18469 raft_consensus.cc:729] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 514bb6c484bd4dd7b61789115254452a, State: Initialized, Role: FOLLOWER
I20260928 21:16:03.408572 18472 ts_tablet_manager.cc:1397] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:03.408913 18469 consensus_queue.cc:260] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.409008 18468 raft_consensus.cc:348] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.409324 18468 raft_consensus.cc:374] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:03.409397 18468 raft_consensus.cc:729] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c17c02162144c4d8793a10ad3fb0460, State: Initialized, Role: FOLLOWER
I20260928 21:16:03.409720 18320 heartbeater.cc:503] Master 127.0.98.62:34651 was elected leader, sending a full tablet report...
I20260928 21:16:03.409904 18468 consensus_queue.cc:260] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.410804 18451 heartbeater.cc:503] Master 127.0.98.62:34651 was elected leader, sending a full tablet report...
I20260928 21:16:03.410974 18468 ts_tablet_manager.cc:1428] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 21:16:03.411466 18469 ts_tablet_manager.cc:1428] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:16:03.413480 18472 raft_consensus.cc:348] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.413793 18472 raft_consensus.cc:374] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:03.413864 18472 raft_consensus.cc:729] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219fd7adecb042c5991542e1a16f88e3, State: Initialized, Role: FOLLOWER
W20260928 21:16:03.413910 18321 tablet.cc:2369] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:03.414330 18472 consensus_queue.cc:260] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.415261 18472 ts_tablet_manager.cc:1428] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:16:03.415673 18189 heartbeater.cc:503] Master 127.0.98.62:34651 was elected leader, sending a full tablet report...
I20260928 21:16:03.433221 18474 raft_consensus.cc:482] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:03.433471 18474 raft_consensus.cc:504] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.434707 18474 leader_election.cc:288] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c17c02162144c4d8793a10ad3fb0460 (127.0.98.3:37897), 219fd7adecb042c5991542e1a16f88e3 (127.0.98.1:45625)
I20260928 21:16:03.438946 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.434952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:37897 (local address 127.0.98.2:49477)
0928 21:16:03.435138 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.435157 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.435273 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.435638 (+ 365us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.435641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.435659 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:03.435873 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.435881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.436742 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.436746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.437739 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.437749 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.438454 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.438458 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.438573 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.438600 (+ 27us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.438689 (+ 89us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.438769 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"spinlock_wait_cycles":18432}
I20260928 21:16:03.438969 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.435137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49477 (local address 127.0.98.3:37897)
0928 21:16:03.435346 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.435353 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.435372 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:16:03.435437 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.435440 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.435488 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.435614 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:03.436009 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.436625 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.437883 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.438321 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.438453 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.438518 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.438746 (+ 228us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.438838 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260928 21:16:03.439102 18467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.435342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49093 (local address 127.0.98.1:45625)
0928 21:16:03.435481 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.435486 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.435695 (+ 209us) server_negotiation.cc:407] Connection header received
0928 21:16:03.435865 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.435868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.435917 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.435995 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:03.436305 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.436950 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.438030 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.438430 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.438568 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.438626 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.438866 (+ 240us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.438947 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:16:03.439488 18477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.435234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45625 (local address 127.0.98.2:49093)
0928 21:16:03.435605 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.435617 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.435762 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:03.436014 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:03.436017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:03.436032 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:03.436189 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.436196 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.437055 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.437061 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.437902 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:03.437912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.438605 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.438610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:03.438719 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.438739 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:16:03.439266 (+ 527us) client_negotiation.cc:240] Negotiation successful
0928 21:16:03.439350 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"spinlock_wait_cycles":896,"thread_start_us":128,"threads_started":1}
I20260928 21:16:03.439620 18405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a04fda056524d478809888a3fb11583" candidate_uuid: "514bb6c484bd4dd7b61789115254452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c17c02162144c4d8793a10ad3fb0460" is_pre_election: true
I20260928 21:16:03.439993 18405 raft_consensus.cc:2454] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 514bb6c484bd4dd7b61789115254452a in term 0.
I20260928 21:16:03.440104 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a04fda056524d478809888a3fb11583" candidate_uuid: "514bb6c484bd4dd7b61789115254452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219fd7adecb042c5991542e1a16f88e3" is_pre_election: true
I20260928 21:16:03.440438 18143 raft_consensus.cc:2454] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 514bb6c484bd4dd7b61789115254452a in term 0.
I20260928 21:16:03.440680 18209 leader_election.cc:302] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 514bb6c484bd4dd7b61789115254452a, 9c17c02162144c4d8793a10ad3fb0460; no voters:
I20260928 21:16:03.441011 18474 raft_consensus.cc:2781] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:03.441087 18474 raft_consensus.cc:482] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:03.441116 18474 raft_consensus.cc:3037] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:03.442157 18474 raft_consensus.cc:504] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.442508 18474 leader_election.cc:288] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [CANDIDATE]: Term 1 election: Requested vote from peers 9c17c02162144c4d8793a10ad3fb0460 (127.0.98.3:37897), 219fd7adecb042c5991542e1a16f88e3 (127.0.98.1:45625)
I20260928 21:16:03.442938 18405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a04fda056524d478809888a3fb11583" candidate_uuid: "514bb6c484bd4dd7b61789115254452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c17c02162144c4d8793a10ad3fb0460"
I20260928 21:16:03.442948 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a04fda056524d478809888a3fb11583" candidate_uuid: "514bb6c484bd4dd7b61789115254452a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "219fd7adecb042c5991542e1a16f88e3"
I20260928 21:16:03.443046 18405 raft_consensus.cc:3037] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:03.443082 18143 raft_consensus.cc:3037] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:03.444108 18405 raft_consensus.cc:2454] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 514bb6c484bd4dd7b61789115254452a in term 1.
I20260928 21:16:03.444118 18143 raft_consensus.cc:2454] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 514bb6c484bd4dd7b61789115254452a in term 1.
I20260928 21:16:03.444447 18209 leader_election.cc:302] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 219fd7adecb042c5991542e1a16f88e3, 514bb6c484bd4dd7b61789115254452a; no voters:
I20260928 21:16:03.444639 18474 raft_consensus.cc:2781] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:03.444883 18474 raft_consensus.cc:686] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 1 LEADER]: Becoming Leader. State: Replica: 514bb6c484bd4dd7b61789115254452a, State: Running, Role: LEADER
I20260928 21:16:03.445251 18474 consensus_queue.cc:237] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } }
I20260928 21:16:03.448570 18000 catalog_manager.cc:5495] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a reported cstate change: term changed from 0 to 1, leader changed from <none> to 514bb6c484bd4dd7b61789115254452a (127.0.98.2). New cstate: current_term: 1 leader_uuid: "514bb6c484bd4dd7b61789115254452a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "514bb6c484bd4dd7b61789115254452a" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45337 } health_report { overall_health: HEALTHY } } }
I20260928 21:16:03.463793 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.460458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42986 (local address 127.0.98.2:45337)
0928 21:16:03.460582 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:03.460586 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:03.460599 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:16:03.460704 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:03.460706 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:03.460760 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:03.460867 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:03.461104 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.461608 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.462435 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:03.462642 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:03.462719 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:03.462994 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:03.463421 (+ 427us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:03.463511 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:03.463560 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 21:16:03.463609 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 21:16:03.467218 18190 tablet.cc:2369] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:03.469875 18143 raft_consensus.cc:1261] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 1 FOLLOWER]: Refusing update from remote peer 514bb6c484bd4dd7b61789115254452a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:03.469936 18405 raft_consensus.cc:1261] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 1 FOLLOWER]: Refusing update from remote peer 514bb6c484bd4dd7b61789115254452a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:03.470597 18474 consensus_queue.cc:1035] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [LEADER]: Connected to new peer: Peer: permanent_uuid: "219fd7adecb042c5991542e1a16f88e3" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:03.470824 18478 consensus_queue.cc:1035] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c17c02162144c4d8793a10ad3fb0460" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 37897 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:03.479090 18482 mvcc.cc:205] Tried to move back new op lower bound from 7334421149565587456 to 7334421149474545664. Current Snapshot: MvccSnapshot[applied={T|T < 7334421149565587456}]
I20260928 21:16:03.479146 18483 mvcc.cc:205] Tried to move back new op lower bound from 7334421149565587456 to 7334421149474545664. Current Snapshot: MvccSnapshot[applied={T|T < 7334421149565587456}]
I20260928 21:16:03.486142 17999 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:03.486325 17999 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50914:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:03.488902 17999 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 88c7da9f5c7b4b1182884eac10996675: Sending DeleteTablet for 3 replicas of tablet 4a04fda056524d478809888a3fb11583
I20260928 21:16:03.489784 18123 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a04fda056524d478809888a3fb11583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:03 UTC) from {username='slave'} at 127.0.0.1:54898
I20260928 21:16:03.489778 18254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a04fda056524d478809888a3fb11583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:03 UTC) from {username='slave'} at 127.0.0.1:42976
I20260928 21:16:03.490043 18385 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a04fda056524d478809888a3fb11583 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:03 UTC) from {username='slave'} at 127.0.0.1:49926
I20260928 21:16:03.490299 18490 tablet_replica.cc:330] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3: stopping tablet replica
I20260928 21:16:03.490653 18492 tablet_replica.cc:330] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: stopping tablet replica
I20260928 21:16:03.490649 18490 raft_consensus.cc:2229] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:03.490988 18490 raft_consensus.cc:2258] T 4a04fda056524d478809888a3fb11583 P 219fd7adecb042c5991542e1a16f88e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:03.490998 18492 raft_consensus.cc:2229] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:03.491230 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18061
I20260928 21:16:03.491266 18492 raft_consensus.cc:2258] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:03.492767 18491 tablet_replica.cc:330] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: stopping tablet replica
I20260928 21:16:03.493108 18491 raft_consensus.cc:2229] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:03.493517 18491 raft_consensus.cc:2258] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:03.494910 18492 ts_tablet_manager.cc:1905] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:03.494926 18491 ts_tablet_manager.cc:1905] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:03.497816 18492 ts_tablet_manager.cc:1918] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:03.497931 18492 log.cc:1196] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/4a04fda056524d478809888a3fb11583
I20260928 21:16:03.498246 18492 ts_tablet_manager.cc:1939] T 4a04fda056524d478809888a3fb11583 P 9c17c02162144c4d8793a10ad3fb0460: Deleting consensus metadata
I20260928 21:16:03.498554 18491 ts_tablet_manager.cc:1918] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:03.498667 18491 log.cc:1196] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/4a04fda056524d478809888a3fb11583
W20260928 21:16:03.499660 17987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45625 (error 108)
I20260928 21:16:03.499809 18491 ts_tablet_manager.cc:1939] T 4a04fda056524d478809888a3fb11583 P 514bb6c484bd4dd7b61789115254452a: Deleting consensus metadata
I20260928 21:16:03.500244 17987 catalog_manager.cc:4841] TS 9c17c02162144c4d8793a10ad3fb0460 (127.0.98.3:37897): tablet 4a04fda056524d478809888a3fb11583 (table test-table [id=11cb4f36b4494656a730b7deaaae17e9]) successfully deleted
W20260928 21:16:03.500969 18464 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:03.500601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45625 (local address 127.0.0.1:54906)
0928 21:16:03.500719 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.500738 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:03.500817 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45625: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":35}
I20260928 21:16:03.501139 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18192
W20260928 21:16:03.501150 17987 catalog_manager.cc:4584] TS 219fd7adecb042c5991542e1a16f88e3 (127.0.98.1:45625): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a04fda056524d478809888a3fb11583: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45625: BlockingWrite error: write error: Connection reset by peer (error 104)
I20260928 21:16:03.506543 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18323
I20260928 21:16:03.507854 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:03.507459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45337 (local address 127.0.0.1:42994)
0928 21:16:03.507664 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:03.507748 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45337: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260928 21:16:03.512244 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 17970
2026-09-28T21:16:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (988 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:03 dist-test-slave-v7xz krb5kdc[18496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:03 dist-test-slave-v7xz krb5kdc[18496](info): set up 2 sockets
Sep 28 21:16:03 dist-test-slave-v7xz krb5kdc[18496](info): commencing operation
krb5kdc: starting...
W20260928 21:16:04.612697 392 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:04 dist-test-slave-v7xz krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:04.737069 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35979
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:04.848150 18509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:04.848487 18509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:04.848570 18509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:04.852797 18509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:04.852898 18509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:04.852937 18509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:04.852996 18509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:04.853036 18509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:04.858366 18509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:04.859476 18509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:04.860533 18509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:04.869174 18517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:04.869190 18515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:04.869233 18514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:04.869238 18509 server_base.cc:1034] running on GCE node
I20260928 21:16:04.869974 18509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:04.871193 18509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:04.872359 18509 hybrid_clock.cc:648] HybridClock initialized: now 1790630164872345 us; error 28 us; skew 500 ppm
Sep 28 21:16:04 dist-test-slave-v7xz krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630164, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:04.875180 18509 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:04.876121 18509 webserver.cc:460] Webserver started at http://127.0.98.62:44415/ using document root <none> and password file <none>
I20260928 21:16:04.876677 18509 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:04.876754 18509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:04.877024 18509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:04.879066 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac"
format_stamp: "Formatted at 2026-09-28 21:16:04 on dist-test-slave-v7xz"
server_key: "352d9c9b2a49125c8f2a5fb926b1ffac"
server_key_iv: "b9d3e40d12e14b0371b2d0a7e2043d11"
server_key_version: "encryptionkey@0"
I20260928 21:16:04.879634 18509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac"
format_stamp: "Formatted at 2026-09-28 21:16:04 on dist-test-slave-v7xz"
server_key: "352d9c9b2a49125c8f2a5fb926b1ffac"
server_key_iv: "b9d3e40d12e14b0371b2d0a7e2043d11"
server_key_version: "encryptionkey@0"
I20260928 21:16:04.883508 18509 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:16:04.886036 18524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:04.887319 18509 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:04.887463 18509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac"
format_stamp: "Formatted at 2026-09-28 21:16:04 on dist-test-slave-v7xz"
server_key: "352d9c9b2a49125c8f2a5fb926b1ffac"
server_key_iv: "b9d3e40d12e14b0371b2d0a7e2043d11"
server_key_version: "encryptionkey@0"
I20260928 21:16:04.887612 18509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:04.903792 18509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:04.906950 18509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:04.907166 18509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:04.914215 18509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35979
I20260928 21:16:04.914309 18576 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35979 every 8 connection(s)
I20260928 21:16:04.915226 18509 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:04.918267 18577 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:04.923696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18509
I20260928 21:16:04.923863 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:04.924213 392 external_mini_cluster.cc:1428] Setting key 1f07b6b100633876a50075930c9bd586
I20260928 21:16:04.924748 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac: Bootstrap starting.
I20260928 21:16:04.927374 18577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:04.928428 18577 log.cc:824] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:04.930910 18577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac: No bootstrap required, opened a new log
Sep 28 21:16:04 dist-test-slave-v7xz krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:04.936228 18577 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35979 } }
I20260928 21:16:04.936545 18577 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:04.936637 18577 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cf67aa07c0f49a090b4b2cbeb9d6bac, State: Initialized, Role: FOLLOWER
I20260928 21:16:04.937197 18577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35979 } }
I20260928 21:16:04.937403 18577 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:04.937485 18577 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:04.937618 18577 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:04.938854 18577 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35979 } }
I20260928 21:16:04.939294 18577 leader_election.cc:302] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5cf67aa07c0f49a090b4b2cbeb9d6bac; no voters:
I20260928 21:16:04.939706 18577 leader_election.cc:288] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:04.939815 18582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:04.940042 18582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [term 1 LEADER]: Becoming Leader. State: Replica: 5cf67aa07c0f49a090b4b2cbeb9d6bac, State: Running, Role: LEADER
I20260928 21:16:04.940395 18582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35979 } }
I20260928 21:16:04.940778 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:04.925695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34592 (local address 127.0.98.62:35979)
0928 21:16:04.926135 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:04.926152 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:04.926189 (+ 37us) server_negotiation.cc:407] Connection header received
0928 21:16:04.926855 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:04.926872 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:04.927218 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:04.927603 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:04.928620 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:04.929728 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:04.931037 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:04.931566 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:04.931692 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:04.934984 (+ 3292us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:04.934990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:04.935006 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:04.935038 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:04.937165 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:04.937756 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:04.937762 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:04.937768 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:04.937902 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:04.938230 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:04.938233 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:04.938254 (+ 21us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:04.938679 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:04.938842 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:04.939221 (+ 379us) server_negotiation.cc:299] Negotiation successful
0928 21:16:04.939619 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":137,"threads_started":1}
I20260928 21:16:04.941283 18577 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:04.941701 18582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5cf67aa07c0f49a090b4b2cbeb9d6bac. Latest consensus state: current_term: 1 leader_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35979 } } }
I20260928 21:16:04.941823 18582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:04.942190 18586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:04.943964 18583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5cf67aa07c0f49a090b4b2cbeb9d6bac" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35979 } } }
I20260928 21:16:04.944082 18583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:04.946064 18586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:04.952190 18586 catalog_manager.cc:1347] Generated new cluster ID: 28c19c202a3640ae872ced5e98979a48
I20260928 21:16:04.952288 18586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:04.965520 18586 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:04.966490 18586 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:04.976408 18586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac: Generated new TSK 0
I20260928 21:16:04.977308 18586 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:16:05.011432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35979
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:05.122675 18603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:05.122956 18603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:05.122999 18603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:05.127146 18603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:05.127203 18603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:05.127260 18603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:16:05.132453 18603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18603
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:05.133544 18603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:05.134558 18603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:05.143939 18609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:05.143930 18611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:05.143953 18608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:05.144183 18603 server_base.cc:1034] running on GCE node
I20260928 21:16:05.144701 18603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:05.145363 18603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:05.146519 18603 hybrid_clock.cc:648] HybridClock initialized: now 1790630165146506 us; error 25 us; skew 500 ppm
Sep 28 21:16:05 dist-test-slave-v7xz krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:05.154244 18603 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:16:05.155179 18603 webserver.cc:460] Webserver started at http://127.0.98.1:36201/ using document root <none> and password file <none>
I20260928 21:16:05.155746 18603 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:05.155807 18603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:05.156039 18603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:05.158115 18603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f3b7c4d40f04536b0ee916effab6954"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "4fd38d3d05b225811832e194b38ae982"
server_key_iv: "e8e9fd6742fd500b3e24a677fa2bc205"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.158675 18603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f3b7c4d40f04536b0ee916effab6954"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "4fd38d3d05b225811832e194b38ae982"
server_key_iv: "e8e9fd6742fd500b3e24a677fa2bc205"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.163100 18603 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260928 21:16:05.165831 18618 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.167052 18603 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:05.167193 18603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "8f3b7c4d40f04536b0ee916effab6954"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "4fd38d3d05b225811832e194b38ae982"
server_key_iv: "e8e9fd6742fd500b3e24a677fa2bc205"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.167318 18603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:05.181644 18603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:05.184973 18603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:05.185189 18603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:05.185801 18603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:05.186652 18603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:05.186730 18603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.186803 18603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:05.186854 18603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.195574 18603 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37681
I20260928 21:16:05.195593 18731 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37681 every 8 connection(s)
I20260928 21:16:05.196452 18603 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:05.197779 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18603
I20260928 21:16:05.197912 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:05.198272 392 external_mini_cluster.cc:1428] Setting key 65f9a7172f980fab3218cbbe99a0c3a8
Sep 28 21:16:05 dist-test-slave-v7xz krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:16:05.211720 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.198373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56301 (local address 127.0.98.62:35979)
0928 21:16:05.198518 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.198522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.199609 (+ 1087us) server_negotiation.cc:407] Connection header received
0928 21:16:05.201466 (+ 1857us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.201468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.201517 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.201593 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:05.202832 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.203362 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.204604 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.204909 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.205022 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.207869 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:05.207872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:05.207874 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:05.207900 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:05.209489 (+ 1589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.210054 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.210056 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.210057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.210133 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.210561 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.210564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.210565 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.210744 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:05.210857 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.211386 (+ 529us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.211536 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:16:05.212934 18734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.198867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35979 (local address 127.0.98.1:56301)
0928 21:16:05.199411 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.199469 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.201229 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.201828 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.201838 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.202208 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:05.202681 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.202697 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.203486 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.203489 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.204476 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.204486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.205061 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.205065 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.205245 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.205854 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:05.205871 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:05.207619 (+ 1748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:05.209619 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:05.209627 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:05.209652 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:05.209953 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:05.210256 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:05.210258 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:05.210259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:05.210466 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:05.210859 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:05.210869 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:05.211141 (+ 272us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.211346 (+ 205us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.211683 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":148,"threads_started":1}
I20260928 21:16:05.214251 18732 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35979
I20260928 21:16:05.214490 18732 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:05.214975 18732 heartbeater.cc:511] Master 127.0.98.62:35979 requested a full tablet report, sending...
I20260928 21:16:05.216468 18541 ts_manager.cc:194] Registered new tserver with Master: 8f3b7c4d40f04536b0ee916effab6954 (127.0.98.1:37681)
I20260928 21:16:05.217679 18541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56301
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:16:05.227723 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35979
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:05.337672 18737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:05.337953 18737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:05.337998 18737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:05.342377 18737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:05.342433 18737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:05.342530 18737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:16:05.347918 18737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:05.349017 18737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:05.350044 18737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:05.360747 18742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:05.360790 18743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:05.361001 18745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:05.361220 18737 server_base.cc:1034] running on GCE node
I20260928 21:16:05.361649 18737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:05.362336 18737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:05.363488 18737 hybrid_clock.cc:648] HybridClock initialized: now 1790630165363474 us; error 26 us; skew 500 ppm
Sep 28 21:16:05 dist-test-slave-v7xz krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:05.366293 18737 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:16:05.367149 18737 webserver.cc:460] Webserver started at http://127.0.98.2:32887/ using document root <none> and password file <none>
I20260928 21:16:05.367704 18737 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:05.367762 18737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:05.367995 18737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:05.370102 18737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "4a97e8974b9844e6b10d8db7515de145"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "4d8dd62c9b383d32e429213d709baea5"
server_key_iv: "e0ecedd5d9010278fbf70e0df5e5e7a8"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.370734 18737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4a97e8974b9844e6b10d8db7515de145"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "4d8dd62c9b383d32e429213d709baea5"
server_key_iv: "e0ecedd5d9010278fbf70e0df5e5e7a8"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.375046 18737 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:16:05.377712 18752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.378958 18737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:05.379118 18737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "4a97e8974b9844e6b10d8db7515de145"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "4d8dd62c9b383d32e429213d709baea5"
server_key_iv: "e0ecedd5d9010278fbf70e0df5e5e7a8"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.379231 18737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:05.402513 18737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:05.405867 18737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:05.406075 18737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:05.406651 18737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:05.407454 18737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:05.407526 18737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.407598 18737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:05.407648 18737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.416291 18737 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32773
I20260928 21:16:05.416319 18865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32773 every 8 connection(s)
I20260928 21:16:05.417201 18737 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:05.424356 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18737
I20260928 21:16:05.424533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:05.424897 392 external_mini_cluster.cc:1428] Setting key 67a7fc06b1121718ce030b175ab1848f
Sep 28 21:16:05 dist-test-slave-v7xz krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:05.431650 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.419125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42935 (local address 127.0.98.62:35979)
0928 21:16:05.419273 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.419278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.420194 (+ 916us) server_negotiation.cc:407] Connection header received
0928 21:16:05.421466 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.421468 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.421517 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.421590 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:05.422933 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.423474 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.424805 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.425169 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.425286 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.427856 (+ 2570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:05.427859 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:05.427861 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:05.427886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:05.429300 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.429926 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.429929 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.429930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.430005 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.430449 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.430452 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.430453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.430640 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:05.430744 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.431428 (+ 684us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.431534 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:16:05.433018 18868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.419556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35979 (local address 127.0.98.2:42935)
0928 21:16:05.420035 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.420081 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.421278 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.421750 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.421757 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.422153 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:05.422634 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.422648 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.423605 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.423607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.424686 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.424696 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.425383 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.425388 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.425594 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.426164 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:05.426179 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:05.427673 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:05.429420 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:05.429429 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:05.429448 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:05.429820 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:05.430114 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:05.430120 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:05.430126 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:05.430319 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:05.430774 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:05.430784 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:05.431146 (+ 362us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.431376 (+ 230us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.431773 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":109,"threads_started":1}
I20260928 21:16:05.434187 18866 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35979
I20260928 21:16:05.434449 18866 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:05.435143 18866 heartbeater.cc:511] Master 127.0.98.62:35979 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260928 21:16:05.436266 18541 ts_manager.cc:194] Registered new tserver with Master: 4a97e8974b9844e6b10d8db7515de145 (127.0.98.2:32773)
I20260928 21:16:05.436954 18541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:16:05.453729 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35979
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:05.562954 18871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:05.563228 18871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:05.563272 18871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:05.567443 18871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:05.567500 18871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:05.567595 18871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:16:05.573132 18871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35979
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:05.574139 18871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:05.575133 18871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:05.584314 18879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:05.584268 18876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:05.584344 18877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:05.584618 18871 server_base.cc:1034] running on GCE node
I20260928 21:16:05.585063 18871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:05.585673 18871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:05.586834 18871 hybrid_clock.cc:648] HybridClock initialized: now 1790630165586816 us; error 29 us; skew 500 ppm
Sep 28 21:16:05 dist-test-slave-v7xz krb5kdc[18496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:05.589484 18871 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:16:05.590320 18871 webserver.cc:460] Webserver started at http://127.0.98.3:41803/ using document root <none> and password file <none>
I20260928 21:16:05.590880 18871 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:05.590940 18871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:05.591096 18871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:05.593140 18871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "bb40a4b273de453f952fc65f05cc71ee"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "e361b38b58e940b536bd5b01fcbb20aa"
server_key_iv: "c399bb2a1bdf8d9bdb06c0daa2b7a0dc"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.593655 18871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "bb40a4b273de453f952fc65f05cc71ee"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "e361b38b58e940b536bd5b01fcbb20aa"
server_key_iv: "c399bb2a1bdf8d9bdb06c0daa2b7a0dc"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.597199 18871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260928 21:16:05.599457 18886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.600627 18871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:05.600760 18871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "bb40a4b273de453f952fc65f05cc71ee"
format_stamp: "Formatted at 2026-09-28 21:16:05 on dist-test-slave-v7xz"
server_key: "e361b38b58e940b536bd5b01fcbb20aa"
server_key_iv: "c399bb2a1bdf8d9bdb06c0daa2b7a0dc"
server_key_version: "encryptionkey@0"
I20260928 21:16:05.600926 18871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:05.618294 18871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:05.621593 18871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:05.621811 18871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:05.622391 18871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:05.623196 18871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:05.623243 18871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.623282 18871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:05.623338 18871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:05.632072 18871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33817
I20260928 21:16:05.632155 18999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33817 every 8 connection(s)
I20260928 21:16:05.633105 18871 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:05.639559 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 18871
I20260928 21:16:05.639671 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:05.639981 392 external_mini_cluster.cc:1428] Setting key c94b99a172c36a9f1c97712bd6910a80
Sep 28 21:16:05 dist-test-slave-v7xz krb5kdc[18496](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630165, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:05.647120 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.634890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44099 (local address 127.0.98.62:35979)
0928 21:16:05.635034 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.635038 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.635925 (+ 887us) server_negotiation.cc:407] Connection header received
0928 21:16:05.637274 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.637277 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.637336 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.637444 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:05.638609 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.639140 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.640645 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.640939 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.641054 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.643776 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:05.643783 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:05.643785 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:05.643810 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:05.645076 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.645595 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.645603 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.645604 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.645681 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.646026 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.646029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.646031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.646224 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:05.646334 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.646833 (+ 499us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.646956 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:16:05.647946 19002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.635327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35979 (local address 127.0.98.3:44099)
0928 21:16:05.635779 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.635822 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.637062 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.637621 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.637628 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.637985 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:05.638459 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.638471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.639314 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.639317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.640500 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.640511 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.641090 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.641094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.641272 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.642011 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:05.642037 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:05.643590 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:05.645191 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:05.645197 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:05.645215 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:05.645477 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:05.645774 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:05.645778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:05.645780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:05.645926 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:05.646331 (+ 405us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:05.646339 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:05.646588 (+ 249us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.646788 (+ 200us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.647123 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":105,"threads_started":1}
I20260928 21:16:05.649135 19000 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35979
I20260928 21:16:05.649380 19000 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:05.649856 19000 heartbeater.cc:511] Master 127.0.98.62:35979 requested a full tablet report, sending...
I20260928 21:16:05.650882 18541 ts_manager.cc:194] Registered new tserver with Master: bb40a4b273de453f952fc65f05cc71ee (127.0.98.3:33817)
I20260928 21:16:05.651423 18541 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44099
I20260928 21:16:05.654062 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:05.663544 18580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.656004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34598 (local address 127.0.98.62:35979)
0928 21:16:05.656129 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.656132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.656340 (+ 208us) server_negotiation.cc:407] Connection header received
0928 21:16:05.656427 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.656430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.656466 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.656532 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:05.657286 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.657813 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.658645 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.658854 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.658932 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.660470 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:05.660473 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:05.660475 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:05.660497 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:05.661953 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.662411 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.662414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.662415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.662483 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:05.662754 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:05.662756 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:05.662758 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:05.662948 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:05.663059 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.663338 (+ 279us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.663421 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:16:05.666642 18541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:05.669590 18541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:05.684280 19014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.678781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32773 (local address 127.0.0.1:45366)
0928 21:16:05.679442 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.679454 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.679533 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.680105 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.680111 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.680133 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:05.680237 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.680244 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.681281 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.681285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.682621 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.682634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.683685 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.683688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.683805 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.683844 (+ 39us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.683982 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.684108 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":92,"threads_started":1}
I20260928 21:16:05.684295 19013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.678459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33817 (local address 127.0.0.1:38262)
0928 21:16:05.679134 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.679148 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.679253 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.680172 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.680175 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.680192 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:05.680284 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.680290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.681740 (+ 1450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.681746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.682890 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.682900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.683962 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.683967 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.684054 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.684074 (+ 20us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.684144 (+ 70us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.684199 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":42,"thread_start_us":93,"threads_started":1}
I20260928 21:16:05.684661 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.678706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38262 (local address 127.0.98.3:33817)
0928 21:16:05.679772 (+ 1066us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.679781 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.679800 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:16:05.679857 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.679861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.680003 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.680176 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:05.680416 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.681555 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.683032 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.683844 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.683950 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.684082 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.684176 (+ 94us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.684290 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":956,"thread_start_us":165,"threads_started":1}
I20260928 21:16:05.684742 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.678038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37681 (local address 127.0.0.1:59186)
0928 21:16:05.678414 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.678467 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.678602 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.679445 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.679451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.679474 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:05.679634 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.679642 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.681639 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.681642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.683136 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.683144 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.684366 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.684371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.684462 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.684483 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.684528 (+ 45us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.684592 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":274,"thread_start_us":93,"threads_started":1}
I20260928 21:16:05.685011 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.678941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.0.98.2:32773)
0928 21:16:05.679347 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.679359 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.679759 (+ 400us) server_negotiation.cc:407] Connection header received
0928 21:16:05.679824 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.679827 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.679967 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.680078 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:05.680362 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.681148 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.682786 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.683568 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.683689 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.683846 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.684685 (+ 839us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.684833 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":93,"threads_started":1}
I20260928 21:16:05.686094 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.678200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.98.1:37681)
0928 21:16:05.678925 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.678939 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.678962 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:05.679039 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.679044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.679265 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.679462 (+ 197us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:05.680263 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.681502 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.683385 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.684221 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.685024 (+ 803us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.685803 (+ 779us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.685855 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.685946 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":106,"threads_started":1}
I20260928 21:16:05.687633 18800 tablet_service.cc:1467] Processing CreateTablet for tablet 96cc3650f2eb464da509735d2b2fe443 (DEFAULT_TABLE table=test-table [id=d99de900fd0046c2a9b3a8fed0c51d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:05.687702 18934 tablet_service.cc:1467] Processing CreateTablet for tablet 96cc3650f2eb464da509735d2b2fe443 (DEFAULT_TABLE table=test-table [id=d99de900fd0046c2a9b3a8fed0c51d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:05.688601 18800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cc3650f2eb464da509735d2b2fe443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:05.688706 18934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cc3650f2eb464da509735d2b2fe443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:05.688582 18666 tablet_service.cc:1467] Processing CreateTablet for tablet 96cc3650f2eb464da509735d2b2fe443 (DEFAULT_TABLE table=test-table [id=d99de900fd0046c2a9b3a8fed0c51d9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:05.689528 18666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 96cc3650f2eb464da509735d2b2fe443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:05.695677 19018 tablet_bootstrap.cc:492] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Bootstrap starting.
I20260928 21:16:05.696178 19017 tablet_bootstrap.cc:492] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Bootstrap starting.
I20260928 21:16:05.697113 19019 tablet_bootstrap.cc:492] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954: Bootstrap starting.
I20260928 21:16:05.698411 19018 tablet_bootstrap.cc:654] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:05.698411 19017 tablet_bootstrap.cc:654] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:05.698907 19019 tablet_bootstrap.cc:654] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:05.699724 19018 log.cc:824] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:05.699724 19017 log.cc:824] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:05.699898 19019 log.cc:824] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:05.701817 19019 tablet_bootstrap.cc:492] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954: No bootstrap required, opened a new log
I20260928 21:16:05.701845 19018 tablet_bootstrap.cc:492] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: No bootstrap required, opened a new log
I20260928 21:16:05.702036 19018 ts_tablet_manager.cc:1397] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260928 21:16:05.702041 19019 ts_tablet_manager.cc:1397] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:16:05.702163 19017 tablet_bootstrap.cc:492] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: No bootstrap required, opened a new log
I20260928 21:16:05.702332 19017 ts_tablet_manager.cc:1397] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:16:05.705257 19018 raft_consensus.cc:348] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.705617 19018 raft_consensus.cc:374] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:05.705705 19018 raft_consensus.cc:729] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb40a4b273de453f952fc65f05cc71ee, State: Initialized, Role: FOLLOWER
I20260928 21:16:05.705966 19019 raft_consensus.cc:348] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.705966 19017 raft_consensus.cc:348] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.706230 19017 raft_consensus.cc:374] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:05.706230 19019 raft_consensus.cc:374] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:05.706158 19018 consensus_queue.cc:260] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.706290 19019 raft_consensus.cc:729] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f3b7c4d40f04536b0ee916effab6954, State: Initialized, Role: FOLLOWER
I20260928 21:16:05.706334 19017 raft_consensus.cc:729] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a97e8974b9844e6b10d8db7515de145, State: Initialized, Role: FOLLOWER
I20260928 21:16:05.706792 19000 heartbeater.cc:503] Master 127.0.98.62:35979 was elected leader, sending a full tablet report...
I20260928 21:16:05.706771 19017 consensus_queue.cc:260] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.706771 19019 consensus_queue.cc:260] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.707171 19018 ts_tablet_manager.cc:1428] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:16:05.707680 19019 ts_tablet_manager.cc:1428] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:16:05.707680 19017 ts_tablet_manager.cc:1428] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260928 21:16:05.707998 18732 heartbeater.cc:503] Master 127.0.98.62:35979 was elected leader, sending a full tablet report...
I20260928 21:16:05.708011 18866 heartbeater.cc:503] Master 127.0.98.62:35979 was elected leader, sending a full tablet report...
I20260928 21:16:05.772001 19024 raft_consensus.cc:482] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:05.772256 19024 raft_consensus.cc:504] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.773527 19024 leader_election.cc:288] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4a97e8974b9844e6b10d8db7515de145 (127.0.98.2:32773), bb40a4b273de453f952fc65f05cc71ee (127.0.98.3:33817)
I20260928 21:16:05.778615 19016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.773873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43477 (local address 127.0.98.2:32773)
0928 21:16:05.774026 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.774031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.774251 (+ 220us) server_negotiation.cc:407] Connection header received
0928 21:16:05.774415 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.774423 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.774475 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.774567 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:05.774906 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.775517 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.777199 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.777896 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.778040 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.778122 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.778389 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.778479 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260928 21:16:05.778615 19015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.774055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34395 (local address 127.0.98.3:33817)
0928 21:16:05.774193 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.774197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.774334 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:16:05.774483 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.774486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.774535 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.774619 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:05.774936 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.775504 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.777153 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.777999 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.778099 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.778160 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.778389 (+ 229us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.778488 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:05.778656 19026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.773745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32773 (local address 127.0.98.1:43477)
0928 21:16:05.774006 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.774021 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.774251 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.774580 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.774583 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.774600 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:05.774780 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.774786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.775656 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.775661 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.777000 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.777015 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.778035 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.778038 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.778165 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.778271 (+ 106us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.778353 (+ 82us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.778469 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"mutex_wait_us":117,"thread_start_us":88,"threads_started":1}
I20260928 21:16:05.778666 19027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.773807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33817 (local address 127.0.98.1:34395)
0928 21:16:05.774236 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.774256 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:05.774373 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:05.774631 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:05.774634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:05.774649 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:05.774824 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.774830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.775629 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.775633 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.776994 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:05.777005 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.778100 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.778103 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:05.778207 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.778274 (+ 67us) client_negotiation.cc:769] Sending connection context
0928 21:16:05.778353 (+ 79us) client_negotiation.cc:240] Negotiation successful
0928 21:16:05.778469 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":144,"thread_start_us":119,"threads_started":1}
I20260928 21:16:05.779325 18954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cc3650f2eb464da509735d2b2fe443" candidate_uuid: "8f3b7c4d40f04536b0ee916effab6954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb40a4b273de453f952fc65f05cc71ee" is_pre_election: true
I20260928 21:16:05.779503 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cc3650f2eb464da509735d2b2fe443" candidate_uuid: "8f3b7c4d40f04536b0ee916effab6954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a97e8974b9844e6b10d8db7515de145" is_pre_election: true
I20260928 21:16:05.779631 18954 raft_consensus.cc:2454] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3b7c4d40f04536b0ee916effab6954 in term 0.
I20260928 21:16:05.779814 18820 raft_consensus.cc:2454] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8f3b7c4d40f04536b0ee916effab6954 in term 0.
I20260928 21:16:05.780086 18622 leader_election.cc:302] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3b7c4d40f04536b0ee916effab6954, bb40a4b273de453f952fc65f05cc71ee; no voters:
I20260928 21:16:05.780388 19024 raft_consensus.cc:2781] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:05.780514 19024 raft_consensus.cc:482] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:05.780571 19024 raft_consensus.cc:3037] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:05.781766 19024 raft_consensus.cc:504] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.782132 19024 leader_election.cc:288] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [CANDIDATE]: Term 1 election: Requested vote from peers 4a97e8974b9844e6b10d8db7515de145 (127.0.98.2:32773), bb40a4b273de453f952fc65f05cc71ee (127.0.98.3:33817)
I20260928 21:16:05.782498 18820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cc3650f2eb464da509735d2b2fe443" candidate_uuid: "8f3b7c4d40f04536b0ee916effab6954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4a97e8974b9844e6b10d8db7515de145"
I20260928 21:16:05.782517 18954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "96cc3650f2eb464da509735d2b2fe443" candidate_uuid: "8f3b7c4d40f04536b0ee916effab6954" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb40a4b273de453f952fc65f05cc71ee"
I20260928 21:16:05.782624 18954 raft_consensus.cc:3037] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:05.782655 18820 raft_consensus.cc:3037] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:05.783747 18954 raft_consensus.cc:2454] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3b7c4d40f04536b0ee916effab6954 in term 1.
I20260928 21:16:05.783789 18820 raft_consensus.cc:2454] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8f3b7c4d40f04536b0ee916effab6954 in term 1.
I20260928 21:16:05.784053 18622 leader_election.cc:302] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f3b7c4d40f04536b0ee916effab6954, bb40a4b273de453f952fc65f05cc71ee; no voters:
I20260928 21:16:05.784258 19024 raft_consensus.cc:2781] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:05.784509 19024 raft_consensus.cc:686] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [term 1 LEADER]: Becoming Leader. State: Replica: 8f3b7c4d40f04536b0ee916effab6954, State: Running, Role: LEADER
I20260928 21:16:05.784879 19024 consensus_queue.cc:237] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } }
I20260928 21:16:05.788000 18540 catalog_manager.cc:5495] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8f3b7c4d40f04536b0ee916effab6954 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8f3b7c4d40f04536b0ee916effab6954" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8f3b7c4d40f04536b0ee916effab6954" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37681 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:05.808465 19012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.804891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.0.98.1:37681)
0928 21:16:05.805024 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:05.805029 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:05.805042 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:16:05.805223 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:05.805226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:05.805294 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:05.805408 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:05.805653 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.806150 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.807015 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:05.807236 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:05.807346 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:05.807650 (+ 304us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:05.808034 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:05.808206 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:05.808291 (+ 85us) server_negotiation.cc:299] Negotiation successful
0928 21:16:05.808329 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:16:05.814131 18820 raft_consensus.cc:1261] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 1 FOLLOWER]: Refusing update from remote peer 8f3b7c4d40f04536b0ee916effab6954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:05.814144 18954 raft_consensus.cc:1261] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 1 FOLLOWER]: Refusing update from remote peer 8f3b7c4d40f04536b0ee916effab6954: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:05.814868 19024 consensus_queue.cc:1035] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb40a4b273de453f952fc65f05cc71ee" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33817 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:05.815075 19028 consensus_queue.cc:1035] T 96cc3650f2eb464da509735d2b2fe443 P 8f3b7c4d40f04536b0ee916effab6954 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4a97e8974b9844e6b10d8db7515de145" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:05.822038 19032 mvcc.cc:205] Tried to move back new op lower bound from 7334421159167352832 to 7334421159057465344. Current Snapshot: MvccSnapshot[applied={T|T < 7334421159167352832}]
I20260928 21:16:05.824258 19033 mvcc.cc:205] Tried to move back new op lower bound from 7334421159167352832 to 7334421159057465344. Current Snapshot: MvccSnapshot[applied={T|T < 7334421159167352832}]
I20260928 21:16:05.830717 18540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:05.830870 18540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:05.833253 18540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5cf67aa07c0f49a090b4b2cbeb9d6bac: Sending DeleteTablet for 3 replicas of tablet 96cc3650f2eb464da509735d2b2fe443
I20260928 21:16:05.834133 18800 tablet_service.cc:1514] Processing DeleteTablet for tablet 96cc3650f2eb464da509735d2b2fe443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45366
I20260928 21:16:05.834167 18666 tablet_service.cc:1514] Processing DeleteTablet for tablet 96cc3650f2eb464da509735d2b2fe443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59186
I20260928 21:16:05.834196 18934 tablet_service.cc:1514] Processing DeleteTablet for tablet 96cc3650f2eb464da509735d2b2fe443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38262
I20260928 21:16:05.834721 19042 tablet_replica.cc:330] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: stopping tablet replica
I20260928 21:16:05.834932 19041 tablet_replica.cc:330] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: stopping tablet replica
I20260928 21:16:05.835287 19041 raft_consensus.cc:2229] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:05.835302 19042 raft_consensus.cc:2229] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:05.835456 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18603
I20260928 21:16:05.835636 19041 raft_consensus.cc:2258] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:05.835723 19042 raft_consensus.cc:2258] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:05.837430 19041 ts_tablet_manager.cc:1905] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:05.839742 19042 ts_tablet_manager.cc:1905] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:05.841087 19041 ts_tablet_manager.cc:1918] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:05.841214 19041 log.cc:1196] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/96cc3650f2eb464da509735d2b2fe443
I20260928 21:16:05.841632 19041 ts_tablet_manager.cc:1939] T 96cc3650f2eb464da509735d2b2fe443 P 4a97e8974b9844e6b10d8db7515de145: Deleting consensus metadata
I20260928 21:16:05.842175 19042 ts_tablet_manager.cc:1918] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:05.842283 19042 log.cc:1196] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/96cc3650f2eb464da509735d2b2fe443
I20260928 21:16:05.842686 19042 ts_tablet_manager.cc:1939] T 96cc3650f2eb464da509735d2b2fe443 P bb40a4b273de453f952fc65f05cc71ee: Deleting consensus metadata
I20260928 21:16:05.843163 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18737
I20260928 21:16:05.843880 18528 catalog_manager.cc:4841] TS bb40a4b273de453f952fc65f05cc71ee (127.0.98.3:33817): tablet 96cc3650f2eb464da509735d2b2fe443 (table test-table [id=d99de900fd0046c2a9b3a8fed0c51d9b]) successfully deleted
I20260928 21:16:05.844576 18527 catalog_manager.cc:4841] TS 4a97e8974b9844e6b10d8db7515de145 (127.0.98.2:32773): tablet 96cc3650f2eb464da509735d2b2fe443 (table test-table [id=d99de900fd0046c2a9b3a8fed0c51d9b]) successfully deleted
W20260928 21:16:05.844857 18527 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37681 (error 108)
I20260928 21:16:05.845955 19011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:05.845602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37681 (local address 127.0.0.1:59208)
0928 21:16:05.845768 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:05.845851 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37681: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260928 21:16:05.846124 18527 catalog_manager.cc:4584] TS 8f3b7c4d40f04536b0ee916effab6954 (127.0.98.1:37681): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 96cc3650f2eb464da509735d2b2fe443: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37681: connect: Connection refused (error 111)
I20260928 21:16:05.849192 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18871
I20260928 21:16:05.854661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 18509
2026-09-28T21:16:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2342 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-28T21:16:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:05Z Disabled control of system clock
I20260928 21:16:05.884263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40621
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:16:05.995327 19047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:05.995604 19047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:05.995640 19047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:05.999822 19047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:05.999881 19047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:05.999895 19047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:05.999948 19047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:05.999963 19047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:06.004747 19047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40621
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40621
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:06.005942 19047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:06.007371 19047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:06.016024 19052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:06.016031 19053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.016153 19047 server_base.cc:1034] running on GCE node
W20260928 21:16:06.016032 19055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.016944 19047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:06.018049 19047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:06.019223 19047 hybrid_clock.cc:648] HybridClock initialized: now 1790630166019204 us; error 34 us; skew 500 ppm
I20260928 21:16:06.021279 19047 webserver.cc:460] Webserver started at http://127.0.98.62:38371/ using document root <none> and password file <none>
I20260928 21:16:06.021881 19047 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:06.021943 19047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:06.022159 19047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:06.024173 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "ab7c413158114bbc88ee22a0c8f2ddce"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "5f5ee198253dd45ad99111187f7da65f"
server_key_iv: "2888c8093ced571d27c6ea70485cf5bc"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.024749 19047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab7c413158114bbc88ee22a0c8f2ddce"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "5f5ee198253dd45ad99111187f7da65f"
server_key_iv: "2888c8093ced571d27c6ea70485cf5bc"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.028421 19047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:06.030885 19061 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.032114 19047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:06.032248 19047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "ab7c413158114bbc88ee22a0c8f2ddce"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "5f5ee198253dd45ad99111187f7da65f"
server_key_iv: "2888c8093ced571d27c6ea70485cf5bc"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.032359 19047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:06.050499 19047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:06.051229 19047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:06.051419 19047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:06.058060 19047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40621
I20260928 21:16:06.058091 19113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40621 every 8 connection(s)
I20260928 21:16:06.059013 19047 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:06.059957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19047
I20260928 21:16:06.060086 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:06.060406 392 external_mini_cluster.cc:1428] Setting key 7574cbb20f17fe70f3bb3b3255578c75
I20260928 21:16:06.062732 19114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:06.068106 19114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce: Bootstrap starting.
I20260928 21:16:06.069356 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.061862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42060 (local address 127.0.98.62:40621)
0928 21:16:06.062464 (+ 602us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.062478 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.062510 (+ 32us) server_negotiation.cc:407] Connection header received
0928 21:16:06.063189 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.063207 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.063587 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.063920 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:06.064343 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.065364 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.066232 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.066684 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.066851 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.067714 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:06.067719 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:06.067731 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:06.068100 (+ 369us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:06.068153 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:06.068192 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:06.068285 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.068393 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.068639 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":125,"threads_started":1}
I20260928 21:16:06.070432 19114 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:06.071148 19114 log.cc:824] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:06.072999 19114 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce: No bootstrap required, opened a new log
I20260928 21:16:06.076277 19114 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260928 21:16:06.076478 19114 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:06.076519 19114 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab7c413158114bbc88ee22a0c8f2ddce, State: Initialized, Role: FOLLOWER
I20260928 21:16:06.076983 19114 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260928 21:16:06.077113 19114 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:06.077257 19114 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:06.077369 19114 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:06.078274 19114 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260928 21:16:06.078552 19114 leader_election.cc:302] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab7c413158114bbc88ee22a0c8f2ddce; no voters:
I20260928 21:16:06.078801 19114 leader_election.cc:288] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:06.078925 19119 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:06.079166 19119 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [term 1 LEADER]: Becoming Leader. State: Replica: ab7c413158114bbc88ee22a0c8f2ddce, State: Running, Role: LEADER
I20260928 21:16:06.079500 19119 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } }
I20260928 21:16:06.080041 19114 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:06.081061 19121 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab7c413158114bbc88ee22a0c8f2ddce. Latest consensus state: current_term: 1 leader_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260928 21:16:06.081197 19121 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:06.081588 19120 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab7c413158114bbc88ee22a0c8f2ddce" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40621 } } }
I20260928 21:16:06.081743 19120 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:06.082188 19128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:06.084640 19128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:06.089936 19128 catalog_manager.cc:1347] Generated new cluster ID: 5395817f11424d629bde178b3c6d2a8c
I20260928 21:16:06.090014 19128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:06.108067 19128 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:06.109158 19128 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:06.118518 19128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce: Generated new TSK 0
I20260928 21:16:06.119292 19128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:06.122354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--builtin_ntp_servers=127.0.98.20:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:16:06.235343 19138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:06.235600 19138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:06.235682 19138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:06.239753 19138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:06.239809 19138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:06.239900 19138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:16:06.244668 19138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:06.245766 19138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:06.247140 19138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:06.256584 19146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:06.256558 19144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.256636 19138 server_base.cc:1034] running on GCE node
W20260928 21:16:06.256558 19143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.257195 19138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:06.257850 19138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:06.259051 19138 hybrid_clock.cc:648] HybridClock initialized: now 1790630166259034 us; error 28 us; skew 500 ppm
I20260928 21:16:06.261004 19138 webserver.cc:460] Webserver started at http://127.0.98.1:35469/ using document root <none> and password file <none>
I20260928 21:16:06.261597 19138 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:06.261694 19138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:06.261921 19138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:06.263898 19138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "2f480ee8c69341e0b05e92e18efca08a"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "6bf558b84aefe0902c087e62813a5b40"
server_key_iv: "e33a4f60cfc95312e197da66f8eb28cf"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.264464 19138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2f480ee8c69341e0b05e92e18efca08a"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "6bf558b84aefe0902c087e62813a5b40"
server_key_iv: "e33a4f60cfc95312e197da66f8eb28cf"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.268139 19138 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 21:16:06.270959 19152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.272181 19138 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:06.272322 19138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "2f480ee8c69341e0b05e92e18efca08a"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "6bf558b84aefe0902c087e62813a5b40"
server_key_iv: "e33a4f60cfc95312e197da66f8eb28cf"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.272440 19138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:06.286940 19138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:06.287638 19138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:06.287833 19138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:06.288410 19138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:06.289392 19138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:06.289463 19138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.289535 19138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:06.289588 19138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.298089 19138 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43433
I20260928 21:16:06.298125 19265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43433 every 8 connection(s)
I20260928 21:16:06.299029 19138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:06.299654 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19138
I20260928 21:16:06.299786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:06.300056 392 external_mini_cluster.cc:1428] Setting key 41df729260c5caba06225448ab10716a
I20260928 21:16:06.302371 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--builtin_ntp_servers=127.0.98.20:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:16:06.310079 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.300984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58307 (local address 127.0.98.62:40621)
0928 21:16:06.301129 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.301133 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.302377 (+ 1244us) server_negotiation.cc:407] Connection header received
0928 21:16:06.304013 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.304016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.304060 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.304135 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:06.305417 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.305934 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.307877 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.308096 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.308174 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.309565 (+ 1391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:06.309567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:06.309569 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:06.309581 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:06.309599 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:06.309606 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:06.309671 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.309877 (+ 206us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.309935 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:16:06.311353 19268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.301515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.1:58307)
0928 21:16:06.302229 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.302271 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.303733 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.304360 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.304370 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.304489 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:06.305258 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.305280 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.306110 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.306116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.307745 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.307761 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.308268 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.308276 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.308518 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.309228 (+ 710us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:06.309233 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:06.309297 (+ 64us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:06.309393 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:06.309704 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:06.309716 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:06.309726 (+ 10us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.309868 (+ 142us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.310211 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":128,"threads_started":1}
I20260928 21:16:06.312678 19266 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260928 21:16:06.312940 19266 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:06.313460 19266 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260928 21:16:06.314986 19078 ts_manager.cc:194] Registered new tserver with Master: 2f480ee8c69341e0b05e92e18efca08a (127.0.98.1:43433)
I20260928 21:16:06.316102 19078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58307
W20260928 21:16:06.417505 19269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:06.417784 19269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:06.417824 19269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:06.423346 19269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:06.423444 19269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:06.423568 19269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:16:06.430402 19269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19269
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:06.431495 19269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:06.432932 19269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:06.442663 19277 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:06.442734 19274 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:06.442682 19275 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.443095 19269 server_base.cc:1034] running on GCE node
I20260928 21:16:06.443459 19269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:06.444110 19269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:06.445273 19269 hybrid_clock.cc:648] HybridClock initialized: now 1790630166445243 us; error 42 us; skew 500 ppm
I20260928 21:16:06.447028 19269 webserver.cc:460] Webserver started at http://127.0.98.2:34271/ using document root <none> and password file <none>
I20260928 21:16:06.447580 19269 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:06.447639 19269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:06.447801 19269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:06.449903 19269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "4954fa819a054eb6a2d602dd497984b6"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "57422f1a1a878882418c42ca500a870f"
server_key_iv: "22a07ddde388be0e6a37d69c6650a9c3"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.450603 19269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4954fa819a054eb6a2d602dd497984b6"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "57422f1a1a878882418c42ca500a870f"
server_key_iv: "22a07ddde388be0e6a37d69c6650a9c3"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.454311 19269 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:06.456691 19283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.457943 19269 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:06.458087 19269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "4954fa819a054eb6a2d602dd497984b6"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "57422f1a1a878882418c42ca500a870f"
server_key_iv: "22a07ddde388be0e6a37d69c6650a9c3"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.458202 19269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:06.475127 19269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:06.475878 19269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:06.476114 19269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:06.476745 19269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:06.477764 19269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:06.477833 19269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.477900 19269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:06.477950 19269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.486892 19269 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41631
I20260928 21:16:06.486919 19396 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41631 every 8 connection(s)
I20260928 21:16:06.487864 19269 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:06.489228 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19269
I20260928 21:16:06.489367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:06.489662 392 external_mini_cluster.cc:1428] Setting key 7d68053030ada2a86ba668e07a20ad25
I20260928 21:16:06.491701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--builtin_ntp_servers=127.0.98.20:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:16:06.498565 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.489844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58699 (local address 127.0.98.62:40621)
0928 21:16:06.490018 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.490025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.491243 (+ 1218us) server_negotiation.cc:407] Connection header received
0928 21:16:06.492732 (+ 1489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.492738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.492805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.492928 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:06.494196 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.495065 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.496212 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.496645 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.496778 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.497935 (+ 1157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:06.497942 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:06.497947 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:06.497966 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:06.497993 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:06.498005 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:06.498123 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.498273 (+ 150us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.498355 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260928 21:16:06.499392 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.490364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.2:58699)
0928 21:16:06.491073 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.491136 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.492517 (+ 1381us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.493164 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.493176 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.493297 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:06.494020 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.494044 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.495193 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.495196 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.496077 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.496095 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.496801 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.496803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.497005 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.497674 (+ 669us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:06.497676 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:06.497714 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:06.497776 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:06.498127 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:06.498135 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:06.498139 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.498241 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.498462 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260928 21:16:06.500617 19397 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260928 21:16:06.500917 19397 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:06.501603 19397 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260928 21:16:06.502790 19078 ts_manager.cc:194] Registered new tserver with Master: 4954fa819a054eb6a2d602dd497984b6 (127.0.98.2:41631)
I20260928 21:16:06.503528 19078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:58699
W20260928 21:16:06.608201 19400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:06.608462 19400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:06.608500 19400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:06.612536 19400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:06.612609 19400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:06.612666 19400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:16:06.617776 19400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40621
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:06.618891 19400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:06.620304 19400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:06.630110 19406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:06.630118 19405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.630264 19400 server_base.cc:1034] running on GCE node
W20260928 21:16:06.630465 19408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:06.630857 19400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:06.631521 19400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:06.632717 19400 hybrid_clock.cc:648] HybridClock initialized: now 1790630166632677 us; error 54 us; skew 500 ppm
I20260928 21:16:06.634625 19400 webserver.cc:460] Webserver started at http://127.0.98.3:45363/ using document root <none> and password file <none>
I20260928 21:16:06.635214 19400 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:06.635309 19400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:06.635565 19400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:06.637720 19400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "8f7df9e721e34a5793274faef0531952"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "e8083aaf706590dfc720089bf636ae24"
server_key_iv: "9cb6d98e1deb35129fae8b0a6e5326c3"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.638336 19400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8f7df9e721e34a5793274faef0531952"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "e8083aaf706590dfc720089bf636ae24"
server_key_iv: "9cb6d98e1deb35129fae8b0a6e5326c3"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.642207 19400 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:16:06.644909 19414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.646191 19400 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:06.646332 19400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "8f7df9e721e34a5793274faef0531952"
format_stamp: "Formatted at 2026-09-28 21:16:06 on dist-test-slave-v7xz"
server_key: "e8083aaf706590dfc720089bf636ae24"
server_key_iv: "9cb6d98e1deb35129fae8b0a6e5326c3"
server_key_version: "encryptionkey@0"
I20260928 21:16:06.646474 19400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:06.666540 19400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:06.667284 19400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:06.667485 19400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:06.668107 19400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:06.669214 19400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:06.669283 19400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.669356 19400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:06.669400 19400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:06.678727 19400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45975
I20260928 21:16:06.678747 19527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45975 every 8 connection(s)
I20260928 21:16:06.679662 19400 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:06.688824 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.681618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47427 (local address 127.0.98.62:40621)
0928 21:16:06.681798 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.681804 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.682893 (+ 1089us) server_negotiation.cc:407] Connection header received
0928 21:16:06.684273 (+ 1380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.684275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.684326 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.684403 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:06.685354 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.685869 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.686994 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.687206 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.687280 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.688328 (+ 1048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:06.688331 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:06.688333 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:06.688346 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:06.688363 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:06.688371 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:06.688428 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.688571 (+ 143us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.688622 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:16:06.688978 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19400
I20260928 21:16:06.689107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:06.689373 392 external_mini_cluster.cc:1428] Setting key c22210855a4fbaf5ed0a22b1dc1c840e
I20260928 21:16:06.689776 19530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.682085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40621 (local address 127.0.98.3:47427)
0928 21:16:06.682735 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.682776 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.684056 (+ 1280us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.684588 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.684594 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.684683 (+ 89us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:06.685227 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.685240 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.685974 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.685977 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.686889 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.686898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.687341 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.687343 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.687488 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.688076 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:06.688078 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:06.688119 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:06.688185 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:06.688443 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:06.688449 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:06.688453 (+ 4us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.688547 (+ 94us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.688773 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260928 21:16:06.691015 19528 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40621
I20260928 21:16:06.691273 19528 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:06.691857 19528 heartbeater.cc:511] Master 127.0.98.62:40621 requested a full tablet report, sending...
I20260928 21:16:06.692864 19078 ts_manager.cc:194] Registered new tserver with Master: 8f7df9e721e34a5793274faef0531952 (127.0.98.3:45975)
I20260928 21:16:06.693454 19078 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:47427
I20260928 21:16:06.703454 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:06.709156 19117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.705513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.0.98.62:40621)
0928 21:16:06.705640 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.705644 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.705799 (+ 155us) server_negotiation.cc:407] Connection header received
0928 21:16:06.705896 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.705898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.705938 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.706003 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:06.706243 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.706737 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.707494 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.707707 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.707781 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.708582 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:06.708585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:06.708587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:06.708599 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:06.708620 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:06.708628 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:06.708754 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.708980 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.709015 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:16:06.712239 19078 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:06.715350 19078 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:06.730530 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.724627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.0.98.2:41631)
0928 21:16:06.725299 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.725311 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.725532 (+ 221us) server_negotiation.cc:407] Connection header received
0928 21:16:06.725768 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.725773 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.725957 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.726078 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:06.726581 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.727470 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.729212 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.729851 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.729983 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.730206 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.730263 (+ 57us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.730358 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":117,"threads_started":1}
I20260928 21:16:06.730518 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.724430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41631 (local address 127.0.0.1:51852)
0928 21:16:06.725433 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.725447 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.725612 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.726220 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.726222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.726236 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:06.726455 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.726461 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.727579 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.727583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.729082 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.729093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.729982 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.729985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.730099 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.730136 (+ 37us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.730238 (+ 102us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.730358 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":943,"mutex_wait_us":127,"thread_start_us":103,"threads_started":1}
I20260928 21:16:06.731077 19543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.725060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45975 (local address 127.0.0.1:53396)
0928 21:16:06.725960 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.725973 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.726047 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.726738 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.726741 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.726759 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:06.726831 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.726837 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.728180 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.728183 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.729314 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.729325 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.730540 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.730543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.730748 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.730796 (+ 48us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.730860 (+ 64us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.730948 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":117,"thread_start_us":86,"threads_started":1}
I20260928 21:16:06.731244 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.724439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59624 (local address 127.0.98.1:43433)
0928 21:16:06.725053 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.725091 (+ 38us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.725270 (+ 179us) server_negotiation.cc:407] Connection header received
0928 21:16:06.725588 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.725594 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.725828 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.725968 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:06.726591 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.727799 (+ 1208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.729370 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.730182 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.730292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.730473 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.730952 (+ 479us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.731054 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505,"thread_start_us":132,"threads_started":1}
I20260928 21:16:06.731534 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.725467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53396 (local address 127.0.98.3:45975)
0928 21:16:06.726297 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.726309 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.726332 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:06.726402 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.726407 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.726600 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.726745 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:06.726952 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.728052 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.729417 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.730182 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.730292 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.730473 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.731268 (+ 795us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.731375 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":185,"threads_started":1}
I20260928 21:16:06.731916 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.724254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43433 (local address 127.0.0.1:59624)
0928 21:16:06.725126 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.725182 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.725389 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.726201 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.726207 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.726230 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:06.726457 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.726464 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.728060 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.728064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.729255 (+ 1191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.729262 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.730540 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.730543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.730752 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.730787 (+ 35us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.730860 (+ 73us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.731767 (+ 907us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":678,"mutex_wait_us":945,"thread_start_us":97,"threads_started":1}
I20260928 21:16:06.734431 19462 tablet_service.cc:1467] Processing CreateTablet for tablet cd76154004bf4728ab2725be7468a633 (DEFAULT_TABLE table=test-table [id=706b8294dab844bca216e2ee6468f0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:06.734331 19331 tablet_service.cc:1467] Processing CreateTablet for tablet cd76154004bf4728ab2725be7468a633 (DEFAULT_TABLE table=test-table [id=706b8294dab844bca216e2ee6468f0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:06.735471 19462 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd76154004bf4728ab2725be7468a633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:06.735477 19331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd76154004bf4728ab2725be7468a633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:06.735319 19200 tablet_service.cc:1467] Processing CreateTablet for tablet cd76154004bf4728ab2725be7468a633 (DEFAULT_TABLE table=test-table [id=706b8294dab844bca216e2ee6468f0e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:06.736336 19200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd76154004bf4728ab2725be7468a633. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:06.741801 19545 tablet_bootstrap.cc:492] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Bootstrap starting.
I20260928 21:16:06.743394 19547 tablet_bootstrap.cc:492] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: Bootstrap starting.
I20260928 21:16:06.743753 19546 tablet_bootstrap.cc:492] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Bootstrap starting.
I20260928 21:16:06.745026 19545 tablet_bootstrap.cc:654] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:06.745628 19547 tablet_bootstrap.cc:654] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:06.746016 19545 log.cc:824] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:06.746454 19546 tablet_bootstrap.cc:654] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:06.746452 19547 log.cc:824] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:06.747174 19546 log.cc:824] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:06.747953 19545 tablet_bootstrap.cc:492] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: No bootstrap required, opened a new log
I20260928 21:16:06.748165 19545 ts_tablet_manager.cc:1397] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20260928 21:16:06.748960 19546 tablet_bootstrap.cc:492] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: No bootstrap required, opened a new log
I20260928 21:16:06.749153 19546 ts_tablet_manager.cc:1397] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20260928 21:16:06.751983 19545 raft_consensus.cc:348] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.752221 19545 raft_consensus.cc:374] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:06.752282 19545 raft_consensus.cc:729] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7df9e721e34a5793274faef0531952, State: Initialized, Role: FOLLOWER
I20260928 21:16:06.752743 19545 consensus_queue.cc:260] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.752874 19546 raft_consensus.cc:348] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.753206 19546 raft_consensus.cc:374] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:06.753278 19546 raft_consensus.cc:729] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4954fa819a054eb6a2d602dd497984b6, State: Initialized, Role: FOLLOWER
I20260928 21:16:06.753630 19528 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260928 21:16:06.753875 19545 ts_tablet_manager.cc:1428] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20260928 21:16:06.754356 19546 consensus_queue.cc:260] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.754572 19552 raft_consensus.cc:482] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:06.754715 19552 raft_consensus.cc:504] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.754817 19547 tablet_bootstrap.cc:492] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: No bootstrap required, opened a new log
I20260928 21:16:06.755023 19547 ts_tablet_manager.cc:1397] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260928 21:16:06.755218 19397 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260928 21:16:06.755410 19546 ts_tablet_manager.cc:1428] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Time spent starting tablet: real 0.006s user 0.003s sys 0.002s
I20260928 21:16:06.757018 19552 leader_election.cc:288] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f480ee8c69341e0b05e92e18efca08a (127.0.98.1:43433), 8f7df9e721e34a5793274faef0531952 (127.0.98.3:45975)
I20260928 21:16:06.760349 19547 raft_consensus.cc:348] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.760635 19547 raft_consensus.cc:374] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:06.760679 19547 raft_consensus.cc:729] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f480ee8c69341e0b05e92e18efca08a, State: Initialized, Role: FOLLOWER
I20260928 21:16:06.761225 19547 consensus_queue.cc:260] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.761909 19266 heartbeater.cc:503] Master 127.0.98.62:40621 was elected leader, sending a full tablet report...
I20260928 21:16:06.762370 19547 ts_tablet_manager.cc:1428] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260928 21:16:06.762820 19399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.757438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43433 (local address 127.0.98.2:36663)
0928 21:16:06.757547 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.757575 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.757656 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.758140 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.758142 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.758161 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:06.758280 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.758288 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.759897 (+ 1609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.759900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.761107 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.761118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.762297 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.762301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.762430 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.762455 (+ 25us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.762529 (+ 74us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.762627 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260928 21:16:06.763006 19554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.757688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45975 (local address 127.0.98.2:41973)
0928 21:16:06.758389 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.758403 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:06.758528 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:06.758856 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:06.758860 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:06.758881 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:06.759000 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.759007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.760215 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.760219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.761409 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:06.761418 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.762394 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.762397 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:06.762492 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.762514 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:16:06.762595 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:16:06.762663 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":638,"thread_start_us":93,"threads_started":1}
I20260928 21:16:06.763005 19544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.758032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41973 (local address 127.0.98.3:45975)
0928 21:16:06.758169 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.758174 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.758466 (+ 292us) server_negotiation.cc:407] Connection header received
0928 21:16:06.758644 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.758647 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.758684 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.758729 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:06.759133 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.759783 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.761586 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.762134 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.762235 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.762292 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.762801 (+ 509us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.762889 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260928 21:16:06.763852 19541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.757576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36663 (local address 127.0.98.1:43433)
0928 21:16:06.757687 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.757692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.757707 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:16:06.757754 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.757757 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.757802 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.757901 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:06.758953 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.759753 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.761393 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.761967 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.762087 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.762149 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.763644 (+ 1495us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.763725 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:16:06.763942 19482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd76154004bf4728ab2725be7468a633" candidate_uuid: "4954fa819a054eb6a2d602dd497984b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7df9e721e34a5793274faef0531952" is_pre_election: true
I20260928 21:16:06.764230 19482 raft_consensus.cc:2454] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4954fa819a054eb6a2d602dd497984b6 in term 0.
I20260928 21:16:06.764432 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd76154004bf4728ab2725be7468a633" candidate_uuid: "4954fa819a054eb6a2d602dd497984b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f480ee8c69341e0b05e92e18efca08a" is_pre_election: true
I20260928 21:16:06.764716 19220 raft_consensus.cc:2454] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4954fa819a054eb6a2d602dd497984b6 in term 0.
I20260928 21:16:06.764767 19287 leader_election.cc:302] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4954fa819a054eb6a2d602dd497984b6, 8f7df9e721e34a5793274faef0531952; no voters:
I20260928 21:16:06.765043 19552 raft_consensus.cc:2781] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:06.765141 19552 raft_consensus.cc:482] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:06.765190 19552 raft_consensus.cc:3037] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:06.766260 19552 raft_consensus.cc:504] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.766644 19552 leader_election.cc:288] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [CANDIDATE]: Term 1 election: Requested vote from peers 2f480ee8c69341e0b05e92e18efca08a (127.0.98.1:43433), 8f7df9e721e34a5793274faef0531952 (127.0.98.3:45975)
I20260928 21:16:06.766954 19220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd76154004bf4728ab2725be7468a633" candidate_uuid: "4954fa819a054eb6a2d602dd497984b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f480ee8c69341e0b05e92e18efca08a"
I20260928 21:16:06.767004 19482 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd76154004bf4728ab2725be7468a633" candidate_uuid: "4954fa819a054eb6a2d602dd497984b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7df9e721e34a5793274faef0531952"
I20260928 21:16:06.767099 19220 raft_consensus.cc:3037] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:06.767150 19482 raft_consensus.cc:3037] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:06.768123 19220 raft_consensus.cc:2454] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4954fa819a054eb6a2d602dd497984b6 in term 1.
I20260928 21:16:06.768151 19482 raft_consensus.cc:2454] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4954fa819a054eb6a2d602dd497984b6 in term 1.
I20260928 21:16:06.768427 19287 leader_election.cc:302] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f480ee8c69341e0b05e92e18efca08a, 4954fa819a054eb6a2d602dd497984b6; no voters:
I20260928 21:16:06.768646 19552 raft_consensus.cc:2781] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:06.768769 19552 raft_consensus.cc:686] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 1 LEADER]: Becoming Leader. State: Replica: 4954fa819a054eb6a2d602dd497984b6, State: Running, Role: LEADER
I20260928 21:16:06.769292 19552 consensus_queue.cc:237] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } }
I20260928 21:16:06.772717 19077 catalog_manager.cc:5495] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4954fa819a054eb6a2d602dd497984b6 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "4954fa819a054eb6a2d602dd497984b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4954fa819a054eb6a2d602dd497984b6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41631 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:06.791214 19542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.787867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.98.2:41631)
0928 21:16:06.788005 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:06.788009 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:06.788021 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:06.788095 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:06.788097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:06.788149 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:06.788238 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:06.788458 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.789022 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.789818 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:06.790013 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:06.790093 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:06.790374 (+ 281us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:06.790760 (+ 386us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:06.790863 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:06.790956 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:16:06.791014 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:16:06.797022 19482 raft_consensus.cc:1261] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 1 FOLLOWER]: Refusing update from remote peer 4954fa819a054eb6a2d602dd497984b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:06.797029 19220 raft_consensus.cc:1261] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a [term 1 FOLLOWER]: Refusing update from remote peer 4954fa819a054eb6a2d602dd497984b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:06.797732 19552 consensus_queue.cc:1035] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7df9e721e34a5793274faef0531952" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:06.797910 19553 consensus_queue.cc:1035] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f480ee8c69341e0b05e92e18efca08a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43433 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260928 21:16:06.800340 19267 tablet.cc:2369] T cd76154004bf4728ab2725be7468a633 P 2f480ee8c69341e0b05e92e18efca08a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:06.816351 19078 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:06.816505 19078 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:06.818851 19078 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab7c413158114bbc88ee22a0c8f2ddce: Sending DeleteTablet for 3 replicas of tablet cd76154004bf4728ab2725be7468a633
I20260928 21:16:06.819717 19200 tablet_service.cc:1514] Processing DeleteTablet for tablet cd76154004bf4728ab2725be7468a633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:06 UTC) from {username='slave'} at 127.0.0.1:59624
I20260928 21:16:06.820219 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19138
I20260928 21:16:06.820278 19462 tablet_service.cc:1514] Processing DeleteTablet for tablet cd76154004bf4728ab2725be7468a633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:06 UTC) from {username='slave'} at 127.0.0.1:53396
I20260928 21:16:06.821305 19331 tablet_service.cc:1514] Processing DeleteTablet for tablet cd76154004bf4728ab2725be7468a633 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:06 UTC) from {username='slave'} at 127.0.0.1:51852
I20260928 21:16:06.821386 19569 tablet_replica.cc:330] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: stopping tablet replica
I20260928 21:16:06.821825 19569 raft_consensus.cc:2229] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:06.822247 19569 raft_consensus.cc:2258] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:06.822620 19570 tablet_replica.cc:330] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: stopping tablet replica
I20260928 21:16:06.822896 19570 raft_consensus.cc:2229] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:06.823278 19570 raft_consensus.cc:2258] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:06.824126 19569 ts_tablet_manager.cc:1905] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:06.824652 19570 ts_tablet_manager.cc:1905] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:06.827276 19570 ts_tablet_manager.cc:1918] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:06.827389 19570 log.cc:1196] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/cd76154004bf4728ab2725be7468a633
I20260928 21:16:06.827703 19570 ts_tablet_manager.cc:1939] T cd76154004bf4728ab2725be7468a633 P 4954fa819a054eb6a2d602dd497984b6: Deleting consensus metadata
I20260928 21:16:06.827827 19569 ts_tablet_manager.cc:1918] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:06.827973 19569 log.cc:1196] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/cd76154004bf4728ab2725be7468a633
I20260928 21:16:06.828351 19569 ts_tablet_manager.cc:1939] T cd76154004bf4728ab2725be7468a633 P 8f7df9e721e34a5793274faef0531952: Deleting consensus metadata
I20260928 21:16:06.828816 19064 catalog_manager.cc:4841] TS 4954fa819a054eb6a2d602dd497984b6 (127.0.98.2:41631): tablet cd76154004bf4728ab2725be7468a633 (table test-table [id=706b8294dab844bca216e2ee6468f0e2]) successfully deleted
W20260928 21:16:06.829437 19065 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43433 (error 108)
I20260928 21:16:06.829670 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19269
I20260928 21:16:06.830291 19065 catalog_manager.cc:4841] TS 8f7df9e721e34a5793274faef0531952 (127.0.98.3:45975): tablet cd76154004bf4728ab2725be7468a633 (table test-table [id=706b8294dab844bca216e2ee6468f0e2]) successfully deleted
I20260928 21:16:06.831228 19539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:06.830954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43433 (local address 127.0.0.1:59628)
0928 21:16:06.831086 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:06.831159 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43433: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 21:16:06.831368 19065 catalog_manager.cc:4584] TS 2f480ee8c69341e0b05e92e18efca08a (127.0.98.1:43433): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cd76154004bf4728ab2725be7468a633: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43433: connect: Connection refused (error 111)
I20260928 21:16:06.835561 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19400
I20260928 21:16:06.841181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19047
2026-09-28T21:16:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (985 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:06 dist-test-slave-v7xz krb5kdc[19574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:06 dist-test-slave-v7xz krb5kdc[19574](info): set up 2 sockets
Sep 28 21:16:06 dist-test-slave-v7xz krb5kdc[19574](info): commencing operation
krb5kdc: starting...
W20260928 21:16:07.880959 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:07 dist-test-slave-v7xz krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:08.007645 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39409
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39409
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:08.119899 19587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:08.120215 19587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:08.120286 19587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:08.124975 19587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:08.125066 19587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:08.125090 19587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:08.125110 19587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:08.125128 19587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:08.131773 19587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39409
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39409
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:08.133056 19587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:08.134179 19587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:08.143146 19595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:08.143172 19592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:08.143134 19593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:08.143347 19587 server_base.cc:1034] running on GCE node
I20260928 21:16:08.144127 19587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:08.145370 19587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:08.146579 19587 hybrid_clock.cc:648] HybridClock initialized: now 1790630168146541 us; error 53 us; skew 500 ppm
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:08.149519 19587 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:08.150467 19587 webserver.cc:460] Webserver started at http://127.0.98.62:33915/ using document root <none> and password file <none>
I20260928 21:16:08.151057 19587 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:08.151150 19587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:08.151388 19587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:08.153635 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "899e987a6bca4e2db4bdae75a3ecdcb3"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "c34c21268149882d23ac8f4b856535a5"
server_key_iv: "7bd8c33f5da7c8bacd0298d5981aab38"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.154206 19587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "899e987a6bca4e2db4bdae75a3ecdcb3"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "c34c21268149882d23ac8f4b856535a5"
server_key_iv: "7bd8c33f5da7c8bacd0298d5981aab38"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.157958 19587 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:16:08.160343 19602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.161769 19587 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:08.161916 19587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "899e987a6bca4e2db4bdae75a3ecdcb3"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "c34c21268149882d23ac8f4b856535a5"
server_key_iv: "7bd8c33f5da7c8bacd0298d5981aab38"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.162041 19587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:08.176918 19587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:08.180312 19587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:08.180533 19587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:08.187757 19587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39409
I20260928 21:16:08.187776 19654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39409 every 8 connection(s)
I20260928 21:16:08.188779 19587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:08.191803 19655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:08.194315 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19587
I20260928 21:16:08.194473 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:08.194741 392 external_mini_cluster.cc:1428] Setting key e9660b0cab63a2070986a561af4f1f8f
I20260928 21:16:08.198261 19655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: Bootstrap starting.
I20260928 21:16:08.200522 19655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:08.201339 19655 log.cc:824] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:08.203339 19655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: No bootstrap required, opened a new log
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630167, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:08.206907 19655 raft_consensus.cc:348] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260928 21:16:08.207149 19655 raft_consensus.cc:374] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:08.207182 19655 raft_consensus.cc:729] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 899e987a6bca4e2db4bdae75a3ecdcb3, State: Initialized, Role: FOLLOWER
I20260928 21:16:08.207710 19655 consensus_queue.cc:260] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260928 21:16:08.207897 19655 raft_consensus.cc:388] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:08.207984 19655 raft_consensus.cc:482] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:08.208096 19655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:08.209385 19655 raft_consensus.cc:504] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260928 21:16:08.209751 19655 leader_election.cc:302] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 899e987a6bca4e2db4bdae75a3ecdcb3; no voters:
I20260928 21:16:08.210120 19655 leader_election.cc:288] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:08.210296 19660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:08.210563 19660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [term 1 LEADER]: Becoming Leader. State: Replica: 899e987a6bca4e2db4bdae75a3ecdcb3, State: Running, Role: LEADER
I20260928 21:16:08.210857 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.196270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.0.98.62:39409)
0928 21:16:08.196748 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:08.196767 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:08.196802 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:16:08.197520 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:08.197537 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:08.197884 (+ 347us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:08.198232 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:08.199052 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.200043 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.201017 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.201526 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.201704 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.204605 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:08.204610 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:08.204625 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:08.204654 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:08.207409 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.208009 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.208015 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.208022 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.208182 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.208480 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.208483 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.208484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.209051 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:08.209268 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:08.209538 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:16:08.209847 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":153,"threads_started":1}
I20260928 21:16:08.210884 19660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } }
I20260928 21:16:08.211414 19655 sys_catalog.cc:564] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:08.213429 19662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 899e987a6bca4e2db4bdae75a3ecdcb3. Latest consensus state: current_term: 1 leader_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } } }
I20260928 21:16:08.213557 19662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:08.214483 19661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "899e987a6bca4e2db4bdae75a3ecdcb3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39409 } } }
I20260928 21:16:08.214586 19661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3 [sys.catalog]: This master's current role is: LEADER
W20260928 21:16:08.216735 19674 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:08.216832 19674 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:08.217242 19675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:08.218264 19675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:08.224242 19675 catalog_manager.cc:1347] Generated new cluster ID: ffd02517c64e44a3a4d5b04c770db3d6
I20260928 21:16:08.224320 19675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:08.236254 19675 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:08.237615 19675 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:08.247923 19675 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: Generated new TSK 0
I20260928 21:16:08.248934 19675 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260928 21:16:08.279752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:08.395167 19681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:08.395467 19681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:08.395542 19681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:08.399745 19681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:08.399829 19681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:08.399899 19681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:16:08.405615 19681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:08.406764 19681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:08.407873 19681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:08.419137 19686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:08.419327 19681 server_base.cc:1034] running on GCE node
W20260928 21:16:08.419157 19689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:08.419147 19687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:08.419821 19681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:08.420575 19681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:08.421774 19681 hybrid_clock.cc:648] HybridClock initialized: now 1790630168421739 us; error 48 us; skew 500 ppm
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:08.424588 19681 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260928 21:16:08.425546 19681 webserver.cc:460] Webserver started at http://127.0.98.1:38215/ using document root <none> and password file <none>
I20260928 21:16:08.426093 19681 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:08.426153 19681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:08.426318 19681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:08.428350 19681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "0393eef5e2dff921651530f943952fc7"
server_key_iv: "fed37f3669ebdd5233d7050ffa747860"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.428947 19681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "0393eef5e2dff921651530f943952fc7"
server_key_iv: "fed37f3669ebdd5233d7050ffa747860"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.433076 19681 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:16:08.435566 19696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.437125 19681 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:08.437269 19681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "0393eef5e2dff921651530f943952fc7"
server_key_iv: "fed37f3669ebdd5233d7050ffa747860"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.437419 19681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:08.480794 19681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:08.484014 19681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:08.484227 19681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:08.484935 19681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:08.485797 19681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:08.485872 19681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.485962 19681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:08.486006 19681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.495028 19681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38485
I20260928 21:16:08.495052 19809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38485 every 8 connection(s)
I20260928 21:16:08.495981 19681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:08.497095 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19681
I20260928 21:16:08.497239 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:08.497524 392 external_mini_cluster.cc:1428] Setting key 29b9c4dfc8f5d30b4f3f1ad369bf05ed
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260928 21:16:08.511665 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.498104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37777 (local address 127.0.98.62:39409)
0928 21:16:08.498249 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:08.498277 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:08.499265 (+ 988us) server_negotiation.cc:407] Connection header received
0928 21:16:08.500167 (+ 902us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:08.500170 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:08.500223 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:08.500302 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:08.501952 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.502483 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.504361 (+ 1878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.504581 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.504677 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.507567 (+ 2890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:08.507571 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:08.507574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:08.507601 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:08.509111 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.509610 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.509613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.509614 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.509690 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.510172 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.510175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.510176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.510355 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:08.510472 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:08.511244 (+ 772us) server_negotiation.cc:299] Negotiation successful
0928 21:16:08.511393 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:16:08.512526 19812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.498551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39409 (local address 127.0.98.1:37777)
0928 21:16:08.499117 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:08.499161 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:08.499958 (+ 797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:08.500483 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:08.500490 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:08.501049 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:08.501765 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:08.501787 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.502622 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.502629 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:08.504215 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:08.504230 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.504752 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.504759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:08.505095 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.505745 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:08.505766 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:08.507396 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:08.509232 (+ 1836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:08.509237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:08.509250 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:08.509519 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:08.509900 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:08.509903 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:08.509904 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:08.510078 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:08.510504 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:08.510510 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:08.510925 (+ 415us) client_negotiation.cc:769] Sending connection context
0928 21:16:08.511166 (+ 241us) client_negotiation.cc:240] Negotiation successful
0928 21:16:08.511472 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":158,"threads_started":1}
I20260928 21:16:08.513988 19810 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39409
I20260928 21:16:08.514253 19810 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:08.514776 19810 heartbeater.cc:511] Master 127.0.98.62:39409 requested a full tablet report, sending...
I20260928 21:16:08.516559 19618 ts_manager.cc:194] Registered new tserver with Master: 2a3fa5c4fdcf40318b37bfbc3c3f4c52 (127.0.98.1:38485)
I20260928 21:16:08.518278 19618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37777
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260928 21:16:08.527626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:08.646626 19815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:08.646922 19815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:08.646970 19815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:08.651043 19815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:08.651114 19815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:08.651175 19815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:16:08.656800 19815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:08.657922 19815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:08.658910 19815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:08.669294 19820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:08.669334 19815 server_base.cc:1034] running on GCE node
W20260928 21:16:08.669299 19821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:08.669355 19823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:08.669960 19815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:08.670655 19815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:08.671890 19815 hybrid_clock.cc:648] HybridClock initialized: now 1790630168671877 us; error 27 us; skew 500 ppm
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:08.674774 19815 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260928 21:16:08.675665 19815 webserver.cc:460] Webserver started at http://127.0.98.2:38155/ using document root <none> and password file <none>
I20260928 21:16:08.676225 19815 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:08.676314 19815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:08.676551 19815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:08.678638 19815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "ef853a47219a4fcb8364ec7e93def99b"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "6bec33189a334b277df73368aedd2e7c"
server_key_iv: "4450d8097b2bd6a0c5c8e35f67008681"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.679190 19815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ef853a47219a4fcb8364ec7e93def99b"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "6bec33189a334b277df73368aedd2e7c"
server_key_iv: "4450d8097b2bd6a0c5c8e35f67008681"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.683050 19815 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:16:08.685595 19830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.686877 19815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:08.687036 19815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "ef853a47219a4fcb8364ec7e93def99b"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "6bec33189a334b277df73368aedd2e7c"
server_key_iv: "4450d8097b2bd6a0c5c8e35f67008681"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.687157 19815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:08.717101 19815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:08.720075 19815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:08.720283 19815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:08.721016 19815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:08.721871 19815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:08.721940 19815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.722012 19815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:08.722066 19815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.731017 19815 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45239
I20260928 21:16:08.731084 19943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45239 every 8 connection(s)
I20260928 21:16:08.732020 19815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:08.734767 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19815
I20260928 21:16:08.734910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:08.735275 392 external_mini_cluster.cc:1428] Setting key 41c61932b019610d57dd194284f70456
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260928 21:16:08.748589 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.734038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50589 (local address 127.0.98.62:39409)
0928 21:16:08.734182 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:08.734186 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:08.735309 (+ 1123us) server_negotiation.cc:407] Connection header received
0928 21:16:08.736282 (+ 973us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:08.736286 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:08.736339 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:08.736477 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:08.737904 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.738671 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.740667 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.741086 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.741205 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.744346 (+ 3141us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:08.744350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:08.744353 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:08.744394 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:08.745947 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.746643 (+ 696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.746647 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.746649 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.746754 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.747135 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.747140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.747142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.747459 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:08.747573 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:08.748266 (+ 693us) server_negotiation.cc:299] Negotiation successful
0928 21:16:08.748416 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:16:08.749565 19946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.734512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39409 (local address 127.0.98.2:50589)
0928 21:16:08.735145 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:08.735193 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:08.736070 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:08.736729 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:08.736740 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:08.737205 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:08.737728 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:08.737743 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.738885 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.738892 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:08.740496 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:08.740512 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.741286 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.741294 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:08.741547 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.742241 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:08.742269 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:08.744145 (+ 1876us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:08.746095 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:08.746102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:08.746122 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:08.746527 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:08.746858 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:08.746861 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:08.746863 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:08.747033 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:08.747574 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:08.747582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:08.747946 (+ 364us) client_negotiation.cc:769] Sending connection context
0928 21:16:08.748221 (+ 275us) client_negotiation.cc:240] Negotiation successful
0928 21:16:08.748607 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":162,"threads_started":1}
I20260928 21:16:08.750938 19944 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39409
I20260928 21:16:08.751226 19944 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:08.751796 19944 heartbeater.cc:511] Master 127.0.98.62:39409 requested a full tablet report, sending...
I20260928 21:16:08.753042 19618 ts_manager.cc:194] Registered new tserver with Master: ef853a47219a4fcb8364ec7e93def99b (127.0.98.2:45239)
I20260928 21:16:08.753638 19618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260928 21:16:08.764370 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:08.876749 19949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:08.877105 19949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:08.877151 19949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:08.881556 19949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:08.881619 19949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:08.881680 19949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:16:08.887009 19949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39409
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:08.888068 19949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:08.889057 19949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:16:08.899426 19949 server_base.cc:1034] running on GCE node
W20260928 21:16:08.899335 19955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:08.899375 19954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:08.899336 19957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:08.900048 19949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:08.900751 19949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:08.901935 19949 hybrid_clock.cc:648] HybridClock initialized: now 1790630168901901 us; error 48 us; skew 500 ppm
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:08.904980 19949 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260928 21:16:08.905925 19949 webserver.cc:460] Webserver started at http://127.0.98.3:43185/ using document root <none> and password file <none>
I20260928 21:16:08.906517 19949 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:08.906610 19949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:08.906852 19949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:08.909031 19949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "7b77544bb1674b90bf5f41b82ab3c4ac"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "e1f64ca742413c942ba9047417aabee2"
server_key_iv: "b4ba51677a56e0071fdd3b5efe0f7f59"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.909618 19949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7b77544bb1674b90bf5f41b82ab3c4ac"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "e1f64ca742413c942ba9047417aabee2"
server_key_iv: "b4ba51677a56e0071fdd3b5efe0f7f59"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.913708 19949 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:16:08.916150 19964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.917446 19949 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:08.917591 19949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "7b77544bb1674b90bf5f41b82ab3c4ac"
format_stamp: "Formatted at 2026-09-28 21:16:08 on dist-test-slave-v7xz"
server_key: "e1f64ca742413c942ba9047417aabee2"
server_key_iv: "b4ba51677a56e0071fdd3b5efe0f7f59"
server_key_version: "encryptionkey@0"
I20260928 21:16:08.917707 19949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:08.950779 19949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:08.953918 19949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:08.954133 19949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:08.954732 19949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:08.955565 19949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:08.955633 19949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.955703 19949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:08.955752 19949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:08.964610 19949 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34381
I20260928 21:16:08.964630 20077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34381 every 8 connection(s)
I20260928 21:16:08.965636 19949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:08.971334 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 19949
I20260928 21:16:08.971527 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:08.971843 392 external_mini_cluster.cc:1428] Setting key cbdc668d686b16be01832e5e3d8094c8
Sep 28 21:16:08 dist-test-slave-v7xz krb5kdc[19574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630168, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:08.979285 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.967565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44195 (local address 127.0.98.62:39409)
0928 21:16:08.967732 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:08.967736 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:08.968761 (+ 1025us) server_negotiation.cc:407] Connection header received
0928 21:16:08.969617 (+ 856us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:08.969620 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:08.969665 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:08.969741 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:08.970879 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.971424 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.972987 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.973291 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.973411 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.976046 (+ 2635us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:08.976049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:08.976051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:08.976075 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:08.977398 (+ 1323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.977859 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.977861 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.977862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.977932 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.978307 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.978309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.978310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.978486 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:08.978560 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:08.979070 (+ 510us) server_negotiation.cc:299] Negotiation successful
0928 21:16:08.979166 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260928 21:16:08.980234 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.967935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39409 (local address 127.0.98.3:44195)
0928 21:16:08.968615 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:08.968657 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:08.969437 (+ 780us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:08.969908 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:08.969915 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:08.970260 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:08.970757 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:08.970770 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.971587 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.971592 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:08.972803 (+ 1211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:08.972813 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.973466 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.973470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:08.973673 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.974406 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:08.974422 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:08.975879 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:08.977502 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:08.977507 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:08.977520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:08.977774 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:08.978039 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:08.978041 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:08.978043 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:08.978222 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:08.978581 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:08.978586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:08.978857 (+ 271us) client_negotiation.cc:769] Sending connection context
0928 21:16:08.979034 (+ 177us) client_negotiation.cc:240] Negotiation successful
0928 21:16:08.979346 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":108,"threads_started":1}
I20260928 21:16:08.981463 20078 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39409
I20260928 21:16:08.981715 20078 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:08.982210 20078 heartbeater.cc:511] Master 127.0.98.62:39409 requested a full tablet report, sending...
I20260928 21:16:08.983340 19618 ts_manager.cc:194] Registered new tserver with Master: 7b77544bb1674b90bf5f41b82ab3c4ac (127.0.98.3:34381)
I20260928 21:16:08.983898 19618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44195
I20260928 21:16:08.986028 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:08.995011 19658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:08.987964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55360 (local address 127.0.98.62:39409)
0928 21:16:08.988087 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:08.988090 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:08.988299 (+ 209us) server_negotiation.cc:407] Connection header received
0928 21:16:08.988391 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:08.988393 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:08.988428 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:08.988488 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:08.989209 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.989705 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.990520 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:08.990733 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:08.990813 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:08.992190 (+ 1377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:08.992192 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:08.992194 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:08.992213 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:08.993507 (+ 1294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.993930 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.993933 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.993934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.994002 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:08.994273 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:08.994275 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:08.994276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:08.994508 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:08.994608 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:08.994805 (+ 197us) server_negotiation.cc:299] Negotiation successful
0928 21:16:08.994895 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:16:08.997954 19618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55360:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:09.001000 19618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:09.015875 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.009492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.1:38485)
0928 21:16:09.009802 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.009819 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.009914 (+ 95us) server_negotiation.cc:407] Connection header received
0928 21:16:09.010171 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.010177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.010408 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.010571 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:09.011292 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.012537 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.014182 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.015046 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.015288 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.015448 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.015611 (+ 163us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.015719 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":100,"threads_started":1}
I20260928 21:16:09.015895 20089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.009382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38485 (local address 127.0.0.1:56654)
0928 21:16:09.009762 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.009841 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:09.009985 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:09.010657 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:09.010662 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:09.010683 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:09.010826 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.010834 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.012664 (+ 1830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.012667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.014074 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.014083 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.015179 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.015183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.015399 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.015451 (+ 52us) client_negotiation.cc:769] Sending connection context
0928 21:16:09.015563 (+ 112us) client_negotiation.cc:240] Negotiation successful
0928 21:16:09.015696 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":45,"thread_start_us":118,"threads_started":1}
I20260928 21:16:09.016297 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.010343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59780 (local address 127.0.98.3:34381)
0928 21:16:09.011154 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.011167 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.011188 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:16:09.011283 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.011288 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.011451 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.011617 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:09.012095 (+ 478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.013119 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.014617 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.015259 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.015693 (+ 434us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.015827 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.015922 (+ 95us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.016118 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":127,"threads_started":1}
I20260928 21:16:09.016539 20091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.010185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34381 (local address 127.0.0.1:59780)
0928 21:16:09.010426 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.010442 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:09.010532 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:09.011574 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:09.011577 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:09.011594 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:09.011710 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.011716 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.013238 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.013243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.014491 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.014502 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.015424 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.015427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.015551 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.015577 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:16:09.015617 (+ 40us) client_negotiation.cc:240] Negotiation successful
0928 21:16:09.016404 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":740,"thread_start_us":98,"threads_started":1}
I20260928 21:16:09.017002 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.010533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45239 (local address 127.0.0.1:46480)
0928 21:16:09.010970 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.010983 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:09.011070 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:09.011831 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:09.011834 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:09.011852 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:09.011931 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.011941 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.013398 (+ 1457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.013401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.015550 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.015562 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.016617 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.016620 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.016707 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.016731 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:16:09.016777 (+ 46us) client_negotiation.cc:240] Negotiation successful
0928 21:16:09.016874 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":1025,"thread_start_us":88,"threads_started":1}
I20260928 21:16:09.017589 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.010877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46480 (local address 127.0.98.2:45239)
0928 21:16:09.011376 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.011388 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.011410 (+ 22us) server_negotiation.cc:407] Connection header received
0928 21:16:09.011485 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.011490 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.011689 (+ 199us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.011832 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:09.012070 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.013238 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.015693 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.016478 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.017172 (+ 694us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.017312 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.017362 (+ 50us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.017447 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":145,"threads_started":1}
I20260928 21:16:09.019292 19744 tablet_service.cc:1467] Processing CreateTablet for tablet 2213485f09d94911bd1cd4502f55026d (DEFAULT_TABLE table=test-table [id=35e33a2757384c78a4244459e2e662ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:09.019655 20012 tablet_service.cc:1467] Processing CreateTablet for tablet 2213485f09d94911bd1cd4502f55026d (DEFAULT_TABLE table=test-table [id=35e33a2757384c78a4244459e2e662ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:09.020347 19744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2213485f09d94911bd1cd4502f55026d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:09.020088 19878 tablet_service.cc:1467] Processing CreateTablet for tablet 2213485f09d94911bd1cd4502f55026d (DEFAULT_TABLE table=test-table [id=35e33a2757384c78a4244459e2e662ff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:09.020678 20012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2213485f09d94911bd1cd4502f55026d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:09.021102 19878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2213485f09d94911bd1cd4502f55026d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:09.027433 20095 tablet_bootstrap.cc:492] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: Bootstrap starting.
I20260928 21:16:09.028553 20096 tablet_bootstrap.cc:492] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Bootstrap starting.
I20260928 21:16:09.029148 20097 tablet_bootstrap.cc:492] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Bootstrap starting.
I20260928 21:16:09.030030 20095 tablet_bootstrap.cc:654] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:09.030826 20095 log.cc:824] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:09.031167 20096 tablet_bootstrap.cc:654] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:09.031219 20097 tablet_bootstrap.cc:654] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:09.032106 20096 log.cc:824] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:09.032385 20097 log.cc:824] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:09.033399 20095 tablet_bootstrap.cc:492] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: No bootstrap required, opened a new log
I20260928 21:16:09.033612 20095 ts_tablet_manager.cc:1397] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:16:09.034168 20096 tablet_bootstrap.cc:492] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: No bootstrap required, opened a new log
I20260928 21:16:09.034308 20097 tablet_bootstrap.cc:492] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: No bootstrap required, opened a new log
I20260928 21:16:09.034364 20096 ts_tablet_manager.cc:1397] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:09.034504 20097 ts_tablet_manager.cc:1397] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260928 21:16:09.037357 20097 raft_consensus.cc:348] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.037632 20097 raft_consensus.cc:374] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:09.037695 20097 raft_consensus.cc:729] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b77544bb1674b90bf5f41b82ab3c4ac, State: Initialized, Role: FOLLOWER
I20260928 21:16:09.038195 20097 consensus_queue.cc:260] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.038385 20095 raft_consensus.cc:348] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.038385 20096 raft_consensus.cc:348] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.038657 20095 raft_consensus.cc:374] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:09.038657 20096 raft_consensus.cc:374] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:09.038717 20096 raft_consensus.cc:729] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef853a47219a4fcb8364ec7e93def99b, State: Initialized, Role: FOLLOWER
I20260928 21:16:09.038726 20095 raft_consensus.cc:729] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a3fa5c4fdcf40318b37bfbc3c3f4c52, State: Initialized, Role: FOLLOWER
I20260928 21:16:09.039028 20078 heartbeater.cc:503] Master 127.0.98.62:39409 was elected leader, sending a full tablet report...
I20260928 21:16:09.039166 20095 consensus_queue.cc:260] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.039166 20096 consensus_queue.cc:260] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.040170 20095 ts_tablet_manager.cc:1428] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260928 21:16:09.040170 20096 ts_tablet_manager.cc:1428] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260928 21:16:09.040428 20097 ts_tablet_manager.cc:1428] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260928 21:16:09.040526 19944 heartbeater.cc:503] Master 127.0.98.62:39409 was elected leader, sending a full tablet report...
I20260928 21:16:09.040516 19810 heartbeater.cc:503] Master 127.0.98.62:39409 was elected leader, sending a full tablet report...
W20260928 21:16:09.216655 20079 tablet.cc:2369] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:09.233218 19945 tablet.cc:2369] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:09.247418 19811 tablet.cc:2369] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:09.278321 20101 raft_consensus.cc:482] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:09.278599 20101 raft_consensus.cc:504] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.279922 20101 leader_election.cc:288] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a3fa5c4fdcf40318b37bfbc3c3f4c52 (127.0.98.1:38485), ef853a47219a4fcb8364ec7e93def99b (127.0.98.2:45239)
I20260928 21:16:09.284198 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.280245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38485 (local address 127.0.98.3:52179)
0928 21:16:09.280385 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.280399 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:09.280501 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:09.280892 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:09.280894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:09.280909 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:09.281055 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.281062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.282028 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.282031 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.282937 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.282946 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.283710 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.283714 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.283867 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.283899 (+ 32us) client_negotiation.cc:769] Sending connection context
0928 21:16:09.283970 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:16:09.284058 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260928 21:16:09.284235 20090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.280375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52179 (local address 127.0.98.1:38485)
0928 21:16:09.280505 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.280509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.280529 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:16:09.280621 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.280626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.280676 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.280759 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:09.281203 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.281905 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.283071 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.283594 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.283678 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.283734 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.284001 (+ 267us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.284084 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260928 21:16:09.284404 20104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.280517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45239 (local address 127.0.98.3:46195)
0928 21:16:09.280878 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.280893 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:09.280988 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:09.281325 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:09.281328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:09.281347 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:09.281470 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.281476 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.282338 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.282340 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.283214 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.283224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.283935 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.283937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.284076 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.284097 (+ 21us) client_negotiation.cc:769] Sending connection context
0928 21:16:09.284178 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:16:09.284242 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"mutex_wait_us":1,"thread_start_us":142,"threads_started":1}
I20260928 21:16:09.284433 20094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.280650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46195 (local address 127.0.98.2:45239)
0928 21:16:09.280789 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.280794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.280985 (+ 191us) server_negotiation.cc:407] Connection header received
0928 21:16:09.281067 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.281071 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.281146 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.281219 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:09.281607 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.282202 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.283352 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.283810 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.283912 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.283991 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.284217 (+ 226us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.284307 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:16:09.285024 19764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2213485f09d94911bd1cd4502f55026d" candidate_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" is_pre_election: true
I20260928 21:16:09.285022 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2213485f09d94911bd1cd4502f55026d" candidate_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef853a47219a4fcb8364ec7e93def99b" is_pre_election: true
I20260928 21:16:09.285333 19898 raft_consensus.cc:2454] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b77544bb1674b90bf5f41b82ab3c4ac in term 0.
I20260928 21:16:09.285336 19764 raft_consensus.cc:2454] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7b77544bb1674b90bf5f41b82ab3c4ac in term 0.
I20260928 21:16:09.285912 19965 leader_election.cc:302] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a3fa5c4fdcf40318b37bfbc3c3f4c52, 7b77544bb1674b90bf5f41b82ab3c4ac; no voters:
I20260928 21:16:09.286247 20101 raft_consensus.cc:2781] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:09.286330 20101 raft_consensus.cc:482] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:09.286383 20101 raft_consensus.cc:3037] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:09.287544 20101 raft_consensus.cc:504] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.287899 20101 leader_election.cc:288] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [CANDIDATE]: Term 1 election: Requested vote from peers 2a3fa5c4fdcf40318b37bfbc3c3f4c52 (127.0.98.1:38485), ef853a47219a4fcb8364ec7e93def99b (127.0.98.2:45239)
I20260928 21:16:09.288486 19764 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2213485f09d94911bd1cd4502f55026d" candidate_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52"
I20260928 21:16:09.288539 19898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2213485f09d94911bd1cd4502f55026d" candidate_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef853a47219a4fcb8364ec7e93def99b"
I20260928 21:16:09.288604 19764 raft_consensus.cc:3037] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:09.288709 19898 raft_consensus.cc:3037] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:09.289780 19764 raft_consensus.cc:2454] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b77544bb1674b90bf5f41b82ab3c4ac in term 1.
I20260928 21:16:09.289799 19898 raft_consensus.cc:2454] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7b77544bb1674b90bf5f41b82ab3c4ac in term 1.
I20260928 21:16:09.290164 19965 leader_election.cc:302] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b77544bb1674b90bf5f41b82ab3c4ac, ef853a47219a4fcb8364ec7e93def99b; no voters:
I20260928 21:16:09.290376 20101 raft_consensus.cc:2781] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:09.290663 20101 raft_consensus.cc:686] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 1 LEADER]: Becoming Leader. State: Replica: 7b77544bb1674b90bf5f41b82ab3c4ac, State: Running, Role: LEADER
I20260928 21:16:09.291071 20101 consensus_queue.cc:237] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } }
I20260928 21:16:09.294329 19618 catalog_manager.cc:5495] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b77544bb1674b90bf5f41b82ab3c4ac (127.0.98.3). New cstate: current_term: 1 leader_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b77544bb1674b90bf5f41b82ab3c4ac" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34381 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:09.308153 20092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.304899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59790 (local address 127.0.98.3:34381)
0928 21:16:09.305017 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.305021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.305033 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:09.305183 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.305185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.305243 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.305337 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:09.305563 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.306047 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.306929 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.307126 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.307198 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.307436 (+ 238us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:09.307844 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:09.307931 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.307983 (+ 52us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.308030 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:16:09.314245 19764 raft_consensus.cc:1261] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 1 FOLLOWER]: Refusing update from remote peer 7b77544bb1674b90bf5f41b82ab3c4ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:09.314253 19898 raft_consensus.cc:1261] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 1 FOLLOWER]: Refusing update from remote peer 7b77544bb1674b90bf5f41b82ab3c4ac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:09.314972 20101 consensus_queue.cc:1035] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a3fa5c4fdcf40318b37bfbc3c3f4c52" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38485 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:09.315174 20105 consensus_queue.cc:1035] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ef853a47219a4fcb8364ec7e93def99b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:09.322336 20110 mvcc.cc:205] Tried to move back new op lower bound from 7334421173503627264 to 7334421173419147264. Current Snapshot: MvccSnapshot[applied={T|T < 7334421173503627264}]
I20260928 21:16:09.331764 19618 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:09.331933 19618 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55360:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:09.334579 19618 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 899e987a6bca4e2db4bdae75a3ecdcb3: Sending DeleteTablet for 3 replicas of tablet 2213485f09d94911bd1cd4502f55026d
I20260928 21:16:09.335472 19744 tablet_service.cc:1514] Processing DeleteTablet for tablet 2213485f09d94911bd1cd4502f55026d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56654
I20260928 21:16:09.335620 19878 tablet_service.cc:1514] Processing DeleteTablet for tablet 2213485f09d94911bd1cd4502f55026d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46480
I20260928 21:16:09.336035 20118 tablet_replica.cc:330] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52: stopping tablet replica
I20260928 21:16:09.336176 20012 tablet_service.cc:1514] Processing DeleteTablet for tablet 2213485f09d94911bd1cd4502f55026d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59780
I20260928 21:16:09.336512 20118 raft_consensus.cc:2229] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:09.336683 20119 tablet_replica.cc:330] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: stopping tablet replica
I20260928 21:16:09.336768 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19681
I20260928 21:16:09.337049 20119 raft_consensus.cc:2229] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:09.337069 20118 raft_consensus.cc:2258] T 2213485f09d94911bd1cd4502f55026d P 2a3fa5c4fdcf40318b37bfbc3c3f4c52 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:09.337392 20119 raft_consensus.cc:2258] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:09.337801 20120 tablet_replica.cc:330] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: stopping tablet replica
I20260928 21:16:09.338112 20120 raft_consensus.cc:2229] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 1 LEADER]: Raft consensus shutting down.
I20260928 21:16:09.338491 20120 raft_consensus.cc:2258] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:09.340314 20120 ts_tablet_manager.cc:1905] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:09.340415 20119 ts_tablet_manager.cc:1905] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:09.343660 20119 ts_tablet_manager.cc:1918] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:09.343789 20119 log.cc:1196] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/2213485f09d94911bd1cd4502f55026d
I20260928 21:16:09.344180 20119 ts_tablet_manager.cc:1939] T 2213485f09d94911bd1cd4502f55026d P ef853a47219a4fcb8364ec7e93def99b: Deleting consensus metadata
I20260928 21:16:09.344537 20120 ts_tablet_manager.cc:1918] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:09.344707 20120 log.cc:1196] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/2213485f09d94911bd1cd4502f55026d
I20260928 21:16:09.345160 20120 ts_tablet_manager.cc:1939] T 2213485f09d94911bd1cd4502f55026d P 7b77544bb1674b90bf5f41b82ab3c4ac: Deleting consensus metadata
I20260928 21:16:09.345638 19603 catalog_manager.cc:4841] TS ef853a47219a4fcb8364ec7e93def99b (127.0.98.2:45239): tablet 2213485f09d94911bd1cd4502f55026d (table test-table [id=35e33a2757384c78a4244459e2e662ff]) successfully deleted
I20260928 21:16:09.346364 19603 catalog_manager.cc:4841] TS 7b77544bb1674b90bf5f41b82ab3c4ac (127.0.98.3:34381): tablet 2213485f09d94911bd1cd4502f55026d (table test-table [id=35e33a2757384c78a4244459e2e662ff]) successfully deleted
W20260928 21:16:09.347232 19603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38485 (error 108)
I20260928 21:16:09.348018 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19815
I20260928 21:16:09.348448 20093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.348061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38485 (local address 127.0.0.1:56666)
0928 21:16:09.348266 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.348342 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
W20260928 21:16:09.348618 19603 catalog_manager.cc:4584] TS 2a3fa5c4fdcf40318b37bfbc3c3f4c52 (127.0.98.1:38485): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2213485f09d94911bd1cd4502f55026d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38485: connect: Connection refused (error 111)
I20260928 21:16:09.354380 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19949
I20260928 21:16:09.359896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 19587
2026-09-28T21:16:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2519 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-28T21:16:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:09Z Disabled control of system clock
I20260928 21:16:09.390511 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:16:09.503300 20125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:09.503598 20125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:09.503634 20125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:09.507879 20125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:09.507952 20125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:09.507968 20125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:09.507985 20125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:09.508001 20125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:09.512953 20125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43283
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:09.514108 20125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:09.515609 20125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:09.524899 20130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:09.524917 20131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:09.524899 20125 server_base.cc:1034] running on GCE node
W20260928 21:16:09.524896 20133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:09.525905 20125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:09.527064 20125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:09.528262 20125 hybrid_clock.cc:648] HybridClock initialized: now 1790630169528247 us; error 31 us; skew 500 ppm
I20260928 21:16:09.530387 20125 webserver.cc:460] Webserver started at http://127.0.98.62:38415/ using document root <none> and password file <none>
I20260928 21:16:09.531005 20125 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:09.531075 20125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:09.531376 20125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:09.533630 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "6a82de4377b246a49491aa81c6960f3b"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "4fef281e7bf4acc1433511e65921393e"
server_key_iv: "d7d1686f192ff045ba44fc85a46c72e7"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.534228 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "6a82de4377b246a49491aa81c6960f3b"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "4fef281e7bf4acc1433511e65921393e"
server_key_iv: "d7d1686f192ff045ba44fc85a46c72e7"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.538007 20125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:09.540581 20139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:09.541905 20125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:16:09.542050 20125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "6a82de4377b246a49491aa81c6960f3b"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "4fef281e7bf4acc1433511e65921393e"
server_key_iv: "d7d1686f192ff045ba44fc85a46c72e7"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.542167 20125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:09.554226 20125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:09.554958 20125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:09.555166 20125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:09.562517 20191 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43283 every 8 connection(s)
I20260928 21:16:09.562515 20125 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43283
I20260928 21:16:09.563587 20125 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:09.566457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20125
I20260928 21:16:09.566593 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:09.566851 20192 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:09.566880 392 external_mini_cluster.cc:1428] Setting key 65c5023451de86eb691f3bcc730b1314
I20260928 21:16:09.574124 20192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b: Bootstrap starting.
I20260928 21:16:09.576996 20192 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:09.577770 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.568313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.0.98.62:43283)
0928 21:16:09.568996 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.569013 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.569058 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:16:09.570259 (+ 1201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.570285 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.570800 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.571166 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:09.571633 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.573157 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.574059 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.574722 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.574875 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.575714 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:09.575721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:09.575737 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:09.576230 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:09.576299 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:09.576325 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:09.576428 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.576525 (+ 97us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.576792 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":109,"threads_started":1}
I20260928 21:16:09.577881 20192 log.cc:824] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:09.580230 20192 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b: No bootstrap required, opened a new log
I20260928 21:16:09.585966 20192 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a82de4377b246a49491aa81c6960f3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260928 21:16:09.586299 20192 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:09.586367 20192 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a82de4377b246a49491aa81c6960f3b, State: Initialized, Role: FOLLOWER
I20260928 21:16:09.586865 20192 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a82de4377b246a49491aa81c6960f3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260928 21:16:09.587056 20192 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:09.587132 20192 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:09.587249 20192 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:09.588553 20192 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a82de4377b246a49491aa81c6960f3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260928 21:16:09.589012 20192 leader_election.cc:302] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a82de4377b246a49491aa81c6960f3b; no voters:
I20260928 21:16:09.589342 20192 leader_election.cc:288] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:09.589593 20197 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:09.589866 20197 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [term 1 LEADER]: Becoming Leader. State: Replica: 6a82de4377b246a49491aa81c6960f3b, State: Running, Role: LEADER
I20260928 21:16:09.590183 20197 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a82de4377b246a49491aa81c6960f3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } }
I20260928 21:16:09.590646 20192 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:09.591854 20199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a82de4377b246a49491aa81c6960f3b. Latest consensus state: current_term: 1 leader_uuid: "6a82de4377b246a49491aa81c6960f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a82de4377b246a49491aa81c6960f3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260928 21:16:09.591995 20199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:09.592278 20198 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a82de4377b246a49491aa81c6960f3b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a82de4377b246a49491aa81c6960f3b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43283 } } }
I20260928 21:16:09.592366 20198 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:09.592497 20206 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:09.595963 20206 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:09.601748 20206 catalog_manager.cc:1347] Generated new cluster ID: 8bcfba5085a248a0917c1c9c1033f429
I20260928 21:16:09.601826 20206 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:09.639536 20206 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:09.640983 20206 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:09.649001 20206 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b: Generated new TSK 0
I20260928 21:16:09.649899 20206 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:09.658319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--builtin_ntp_servers=127.0.98.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260928 21:16:09.769596 20216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:09.769865 20216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:09.769901 20216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:09.773911 20216 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:09.773969 20216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:09.774024 20216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260928 21:16:09.778766 20216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20216
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:09.779799 20216 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:09.781168 20216 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:09.790530 20222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:09.790575 20216 server_base.cc:1034] running on GCE node
W20260928 21:16:09.790555 20221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:09.790515 20224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:09.791306 20216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:09.791996 20216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:09.793179 20216 hybrid_clock.cc:648] HybridClock initialized: now 1790630169793160 us; error 29 us; skew 500 ppm
I20260928 21:16:09.795047 20216 webserver.cc:460] Webserver started at http://127.0.98.1:34199/ using document root <none> and password file <none>
I20260928 21:16:09.795620 20216 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:09.795683 20216 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:09.795924 20216 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:09.798038 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data/instance:
uuid: "83ae8008f1d4447c81688aa37738efea"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "7ec40f8ad323e551e1b4715088bf99dd"
server_key_iv: "008fb3bd2cd526b02f3483ca09085490"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.798606 20216 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal/instance:
uuid: "83ae8008f1d4447c81688aa37738efea"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "7ec40f8ad323e551e1b4715088bf99dd"
server_key_iv: "008fb3bd2cd526b02f3483ca09085490"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.802211 20216 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:09.804685 20230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:09.805984 20216 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:09.806118 20216 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal
uuid: "83ae8008f1d4447c81688aa37738efea"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "7ec40f8ad323e551e1b4715088bf99dd"
server_key_iv: "008fb3bd2cd526b02f3483ca09085490"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.806247 20216 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:09.818666 20216 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:09.819411 20216 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:09.819622 20216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:09.820225 20216 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:09.821254 20216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:09.821339 20216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:09.821430 20216 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:09.821482 20216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:09.830811 20216 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43377
I20260928 21:16:09.830828 20343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43377 every 8 connection(s)
I20260928 21:16:09.831776 20216 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/data/info.pb
I20260928 21:16:09.834830 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20216
I20260928 21:16:09.834964 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-0/wal/instance
I20260928 21:16:09.835318 392 external_mini_cluster.cc:1428] Setting key 54ee25a0f909cf7bcb9e5b7aa295b3f7
I20260928 21:16:09.837698 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--builtin_ntp_servers=127.0.98.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:16:09.842844 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.833711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51043 (local address 127.0.98.62:43283)
0928 21:16:09.833854 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:09.833858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:09.835234 (+ 1376us) server_negotiation.cc:407] Connection header received
0928 21:16:09.836980 (+ 1746us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:09.836983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:09.837039 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:09.837123 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:09.838382 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.839060 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.840221 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.840654 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.840777 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.842232 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:09.842239 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:09.842244 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:09.842262 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:09.842290 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:09.842304 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:09.842388 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:09.842548 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:16:09.842633 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:16:09.843806 20346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:09.834165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.1:51043)
0928 21:16:09.835032 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:09.835094 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:09.836700 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:09.837340 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:09.837349 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:09.837463 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:09.838222 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.838245 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.839201 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.839204 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.840089 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:09.840099 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:09.840803 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:09.840806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:09.841010 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:09.841949 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:09.841950 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:09.841993 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:09.842062 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:09.842403 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:09.842409 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:09.842414 (+ 5us) client_negotiation.cc:769] Sending connection context
0928 21:16:09.842507 (+ 93us) client_negotiation.cc:240] Negotiation successful
0928 21:16:09.842810 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":120,"threads_started":1}
I20260928 21:16:09.845148 20344 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260928 21:16:09.845440 20344 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:09.846189 20344 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260928 21:16:09.847872 20156 ts_manager.cc:194] Registered new tserver with Master: 83ae8008f1d4447c81688aa37738efea (127.0.98.1:43377)
I20260928 21:16:09.849128 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:51043
W20260928 21:16:09.952595 20347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:09.952917 20347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:09.952963 20347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:09.957250 20347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:09.957319 20347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:09.957410 20347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260928 21:16:09.962308 20347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:09.963464 20347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:09.964920 20347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:09.975289 20353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:09.975303 20352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:09.975359 20347 server_base.cc:1034] running on GCE node
W20260928 21:16:09.975283 20355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:09.975996 20347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:09.976665 20347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:09.977865 20347 hybrid_clock.cc:648] HybridClock initialized: now 1790630169977855 us; error 28 us; skew 500 ppm
I20260928 21:16:09.979743 20347 webserver.cc:460] Webserver started at http://127.0.98.2:42231/ using document root <none> and password file <none>
I20260928 21:16:09.980337 20347 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:09.980432 20347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:09.980672 20347 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:09.982954 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data/instance:
uuid: "ce13a5cd4c4945b8b7694b4505884f17"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "8902c8de79373007a881c2578e546984"
server_key_iv: "5507f008d4f1ba118b91277f73e79fec"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.983520 20347 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ce13a5cd4c4945b8b7694b4505884f17"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "8902c8de79373007a881c2578e546984"
server_key_iv: "5507f008d4f1ba118b91277f73e79fec"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.987249 20347 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260928 21:16:09.989730 20361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:09.991019 20347 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:09.991145 20347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal
uuid: "ce13a5cd4c4945b8b7694b4505884f17"
format_stamp: "Formatted at 2026-09-28 21:16:09 on dist-test-slave-v7xz"
server_key: "8902c8de79373007a881c2578e546984"
server_key_iv: "5507f008d4f1ba118b91277f73e79fec"
server_key_version: "encryptionkey@0"
I20260928 21:16:09.991261 20347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:10.016686 20347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:10.017436 20347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:10.017647 20347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:10.018239 20347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:10.019205 20347 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:10.019275 20347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:10.019347 20347 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:10.019399 20347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:10.028628 20347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41943
I20260928 21:16:10.028645 20474 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41943 every 8 connection(s)
I20260928 21:16:10.029659 20347 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/data/info.pb
I20260928 21:16:10.034581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20347
I20260928 21:16:10.034731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal/instance
I20260928 21:16:10.035045 392 external_mini_cluster.cc:1428] Setting key a328e2f4531d1a2d82abe87da47e43ae
I20260928 21:16:10.037135 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--builtin_ntp_servers=127.0.98.20:39897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260928 21:16:10.039624 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.031548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39177 (local address 127.0.98.62:43283)
0928 21:16:10.031693 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.031697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.032784 (+ 1087us) server_negotiation.cc:407] Connection header received
0928 21:16:10.034282 (+ 1498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.034285 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.034333 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.034399 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:10.035650 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.036216 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.037597 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.037825 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.037939 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.039104 (+ 1165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:10.039108 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:10.039111 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:10.039131 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:10.039149 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:10.039158 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:10.039236 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.039405 (+ 169us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.039476 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260928 21:16:10.040663 20477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.031979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.2:39177)
0928 21:16:10.032645 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.032687 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.034065 (+ 1378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.034603 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.034615 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.034744 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:10.035483 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.035501 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.036365 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.036368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.037469 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.037479 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.037994 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.037998 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.038165 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.038820 (+ 655us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:10.038822 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:10.038864 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:10.038940 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:10.039238 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:10.039250 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:10.039258 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.039396 (+ 138us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.039702 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":116,"threads_started":1}
I20260928 21:16:10.042054 20475 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260928 21:16:10.042325 20475 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:10.042855 20475 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260928 21:16:10.043953 20156 ts_manager.cc:194] Registered new tserver with Master: ce13a5cd4c4945b8b7694b4505884f17 (127.0.98.2:41943)
I20260928 21:16:10.044519 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39177
W20260928 21:16:10.153326 20478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:10.153627 20478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:10.153707 20478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:10.157833 20478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:10.157920 20478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:10.158007 20478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260928 21:16:10.162927 20478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:10.164093 20478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:10.165599 20478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:10.175482 20484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:10.175498 20486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:10.175498 20483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:10.175963 20478 server_base.cc:1034] running on GCE node
I20260928 21:16:10.176359 20478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:10.177058 20478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:10.178228 20478 hybrid_clock.cc:648] HybridClock initialized: now 1790630170178208 us; error 32 us; skew 500 ppm
I20260928 21:16:10.193094 20478 webserver.cc:460] Webserver started at http://127.0.98.3:45857/ using document root <none> and password file <none>
I20260928 21:16:10.193746 20478 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:10.193815 20478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:10.193987 20478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:10.196247 20478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data/instance:
uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda"
format_stamp: "Formatted at 2026-09-28 21:16:10 on dist-test-slave-v7xz"
server_key: "b3a11ca4a7fcf695bbda0b62ad0fd37e"
server_key_iv: "f72c43e856345211da622c6d3da5caa7"
server_key_version: "encryptionkey@0"
I20260928 21:16:10.196777 20478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal/instance:
uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda"
format_stamp: "Formatted at 2026-09-28 21:16:10 on dist-test-slave-v7xz"
server_key: "b3a11ca4a7fcf695bbda0b62ad0fd37e"
server_key_iv: "f72c43e856345211da622c6d3da5caa7"
server_key_version: "encryptionkey@0"
I20260928 21:16:10.200503 20478 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:10.203039 20492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:10.204337 20478 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:10.204437 20478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal
uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda"
format_stamp: "Formatted at 2026-09-28 21:16:10 on dist-test-slave-v7xz"
server_key: "b3a11ca4a7fcf695bbda0b62ad0fd37e"
server_key_iv: "f72c43e856345211da622c6d3da5caa7"
server_key_version: "encryptionkey@0"
I20260928 21:16:10.204528 20478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:10.218691 20478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:10.219352 20478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:10.219503 20478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:10.220053 20478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260928 21:16:10.221024 20478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260928 21:16:10.221071 20478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:10.221110 20478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260928 21:16:10.221139 20478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:10.230854 20478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43821
I20260928 21:16:10.230881 20605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43821 every 8 connection(s)
I20260928 21:16:10.231832 20478 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/data/info.pb
I20260928 21:16:10.233682 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20478
I20260928 21:16:10.233815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal/instance
I20260928 21:16:10.234125 392 external_mini_cluster.cc:1428] Setting key 998b368e8dd6dcbf91f021488725f954
I20260928 21:16:10.242126 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.233933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46057 (local address 127.0.98.62:43283)
0928 21:16:10.234074 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.234078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.235395 (+ 1317us) server_negotiation.cc:407] Connection header received
0928 21:16:10.237255 (+ 1860us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.237258 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.237324 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.237405 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:10.238385 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.238911 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.240082 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.240324 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.240441 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.241588 (+ 1147us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:10.241591 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:10.241594 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:10.241608 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:10.241637 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:10.241645 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:10.241726 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.241909 (+ 183us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.241971 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260928 21:16:10.243014 20608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.234465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43283 (local address 127.0.98.3:46057)
0928 21:16:10.235257 (+ 792us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.235312 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.237037 (+ 1725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.237596 (+ 559us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.237604 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.237698 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:10.238198 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.238210 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.239044 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.239048 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.239951 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.239960 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.240481 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.240485 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.240659 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.241316 (+ 657us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0928 21:16:10.241318 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:10.241368 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0928 21:16:10.241433 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:10.241727 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:10.241739 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:10.241747 (+ 8us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.241896 (+ 149us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.242180 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260928 21:16:10.244244 20606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43283
I20260928 21:16:10.244510 20606 heartbeater.cc:464] Registering TS with master...
I20260928 21:16:10.245165 20606 heartbeater.cc:511] Master 127.0.98.62:43283 requested a full tablet report, sending...
I20260928 21:16:10.246227 20156 ts_manager.cc:194] Registered new tserver with Master: 7f7bfee14c544bc8b37adbcd1cfc3cda (127.0.98.3:43821)
I20260928 21:16:10.246768 20156 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:46057
I20260928 21:16:10.248675 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260928 21:16:10.254385 20195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.250651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50280 (local address 127.0.98.62:43283)
0928 21:16:10.250804 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.250808 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.250952 (+ 144us) server_negotiation.cc:407] Connection header received
0928 21:16:10.251069 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.251071 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.251113 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.251181 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:10.251436 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.251933 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.252773 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.253026 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.253117 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.253884 (+ 767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:10.253887 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:10.253889 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:10.253901 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:10.253920 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:10.253928 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:10.253987 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.254141 (+ 154us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.254191 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260928 21:16:10.257730 20156 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50280:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260928 21:16:10.260885 20156 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260928 21:16:10.275345 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.269873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37648 (local address 127.0.98.3:43821)
0928 21:16:10.270217 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.270230 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.270257 (+ 27us) server_negotiation.cc:407] Connection header received
0928 21:16:10.270496 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.270502 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.270747 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.270917 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:10.271425 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.272564 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.274085 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.274648 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.274774 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.274981 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.275074 (+ 93us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.275174 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":110,"threads_started":1}
I20260928 21:16:10.275338 20617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.269692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43821 (local address 127.0.0.1:37648)
0928 21:16:10.270104 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.270163 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.270301 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.271076 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.271084 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.271111 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:10.271272 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.271283 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.272671 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.272677 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.273963 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.273971 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.274774 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.274777 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.274881 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.274917 (+ 36us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.275044 (+ 127us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.275173 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":112,"threads_started":1}
I20260928 21:16:10.276265 20621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.270870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41943 (local address 127.0.0.1:40162)
0928 21:16:10.271399 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.271416 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.271508 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.272201 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.272204 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.272222 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:10.272306 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.272311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.273654 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.273657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.274478 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.274487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.275882 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.275885 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.275970 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.275994 (+ 24us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.276065 (+ 71us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.276132 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":18,"thread_start_us":83,"threads_started":1}
I20260928 21:16:10.276615 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.271296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40162 (local address 127.0.98.2:41943)
0928 21:16:10.271756 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.271765 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.271784 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:16:10.271843 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.271846 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.272051 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.272214 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:10.272405 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.273520 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.274617 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.275757 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.275884 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.276090 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.276139 (+ 49us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.276236 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":117,"threads_started":1}
I20260928 21:16:10.277628 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.270553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43377 (local address 127.0.0.1:33462)
0928 21:16:10.271151 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.271166 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.271243 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.271975 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.271978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.271996 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:10.272119 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.272130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.273654 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.273657 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.275800 (+ 2143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.275811 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.277211 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.277215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.277292 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.277309 (+ 17us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.277407 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.277479 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"mutex_wait_us":4,"thread_start_us":102,"threads_started":1}
I20260928 21:16:10.277751 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.270726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33462 (local address 127.0.98.1:43377)
0928 21:16:10.271518 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.271531 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.271554 (+ 23us) server_negotiation.cc:407] Connection header received
0928 21:16:10.271627 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.271632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.271823 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.271970 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:10.272262 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.273510 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.275941 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.276773 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.276955 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.277152 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.277441 (+ 289us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.277577 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":684,"thread_start_us":128,"threads_started":1}
I20260928 21:16:10.278597 20540 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb4d345a5344a09b5aeb588b3f1967 (DEFAULT_TABLE table=test-table [id=0d0826b0ef854528ba5928f17b96c05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:10.279671 20540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb4d345a5344a09b5aeb588b3f1967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:10.282068 20278 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb4d345a5344a09b5aeb588b3f1967 (DEFAULT_TABLE table=test-table [id=0d0826b0ef854528ba5928f17b96c05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:10.282079 20409 tablet_service.cc:1467] Processing CreateTablet for tablet f2cb4d345a5344a09b5aeb588b3f1967 (DEFAULT_TABLE table=test-table [id=0d0826b0ef854528ba5928f17b96c05e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260928 21:16:10.283113 20278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb4d345a5344a09b5aeb588b3f1967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:10.283129 20409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f2cb4d345a5344a09b5aeb588b3f1967. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:10.287504 20623 tablet_bootstrap.cc:492] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Bootstrap starting.
I20260928 21:16:10.289842 20624 tablet_bootstrap.cc:492] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Bootstrap starting.
I20260928 21:16:10.290630 20623 tablet_bootstrap.cc:654] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:10.290723 20625 tablet_bootstrap.cc:492] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: Bootstrap starting.
I20260928 21:16:10.292059 20623 log.cc:824] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:10.292199 20624 tablet_bootstrap.cc:654] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:10.293078 20624 log.cc:824] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:10.293108 20625 tablet_bootstrap.cc:654] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:10.293884 20625 log.cc:824] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:10.294394 20623 tablet_bootstrap.cc:492] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: No bootstrap required, opened a new log
I20260928 21:16:10.294667 20623 ts_tablet_manager.cc:1397] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260928 21:16:10.295002 20624 tablet_bootstrap.cc:492] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: No bootstrap required, opened a new log
I20260928 21:16:10.295207 20624 ts_tablet_manager.cc:1397] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260928 21:16:10.296123 20625 tablet_bootstrap.cc:492] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: No bootstrap required, opened a new log
I20260928 21:16:10.296300 20625 ts_tablet_manager.cc:1397] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260928 21:16:10.298825 20624 raft_consensus.cc:348] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.299149 20624 raft_consensus.cc:374] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:10.299206 20624 raft_consensus.cc:729] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce13a5cd4c4945b8b7694b4505884f17, State: Initialized, Role: FOLLOWER
I20260928 21:16:10.299702 20624 consensus_queue.cc:260] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.299642 20623 raft_consensus.cc:348] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.299908 20623 raft_consensus.cc:374] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:10.299957 20623 raft_consensus.cc:729] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f7bfee14c544bc8b37adbcd1cfc3cda, State: Initialized, Role: FOLLOWER
I20260928 21:16:10.300416 20623 consensus_queue.cc:260] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.300525 20625 raft_consensus.cc:348] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.300774 20624 ts_tablet_manager.cc:1428] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260928 21:16:10.300794 20625 raft_consensus.cc:374] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:10.300870 20625 raft_consensus.cc:729] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83ae8008f1d4447c81688aa37738efea, State: Initialized, Role: FOLLOWER
I20260928 21:16:10.301239 20475 heartbeater.cc:503] Master 127.0.98.62:43283 was elected leader, sending a full tablet report...
I20260928 21:16:10.301331 20625 consensus_queue.cc:260] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.301501 20623 ts_tablet_manager.cc:1428] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260928 21:16:10.301846 20606 heartbeater.cc:503] Master 127.0.98.62:43283 was elected leader, sending a full tablet report...
I20260928 21:16:10.302341 20625 ts_tablet_manager.cc:1428] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260928 21:16:10.302357 20344 heartbeater.cc:503] Master 127.0.98.62:43283 was elected leader, sending a full tablet report...
W20260928 21:16:10.333022 20345 tablet.cc:2369] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260928 21:16:10.482899 20607 tablet.cc:2369] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:10.519251 20631 raft_consensus.cc:482] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260928 21:16:10.519477 20631 raft_consensus.cc:504] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.520565 20631 leader_election.cc:288] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f7bfee14c544bc8b37adbcd1cfc3cda (127.0.98.3:43821), ce13a5cd4c4945b8b7694b4505884f17 (127.0.98.2:41943)
I20260928 21:16:10.524788 20632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.520837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43821 (local address 127.0.98.1:40601)
0928 21:16:10.521115 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.521130 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.521272 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.521628 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.521631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.521671 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:10.521828 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.521836 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.522566 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.522569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.523543 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.523552 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.524291 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.524295 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.524383 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.524409 (+ 26us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.524512 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.524599 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"spinlock_wait_cycles":1792,"thread_start_us":74,"threads_started":1}
I20260928 21:16:10.524909 20618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.520957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40601 (local address 127.0.98.3:43821)
0928 21:16:10.521134 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.521141 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.521214 (+ 73us) server_negotiation.cc:407] Connection header received
0928 21:16:10.521454 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.521458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.521510 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.521593 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:10.521956 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.522452 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.523664 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.524182 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.524259 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.524307 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.524522 (+ 215us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.524736 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260928 21:16:10.525091 20622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.521411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51747 (local address 127.0.98.2:41943)
0928 21:16:10.521543 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.521546 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.521687 (+ 141us) server_negotiation.cc:407] Connection header received
0928 21:16:10.521826 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.521829 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.521888 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.521962 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0928 21:16:10.522232 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.522874 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.524006 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.524509 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.524605 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.524657 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.524893 (+ 236us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.524977 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:16:10.525235 20633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.521275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41943 (local address 127.0.98.1:51747)
0928 21:16:10.521600 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.521613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:10.521696 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:10.521977 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:10.521980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:10.521995 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0928 21:16:10.522114 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.522120 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.522997 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.522999 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.523874 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:10.523883 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.524629 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.524634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:10.524732 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.524754 (+ 22us) client_negotiation.cc:769] Sending connection context
0928 21:16:10.524852 (+ 98us) client_negotiation.cc:240] Negotiation successful
0928 21:16:10.525075 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":73,"threads_started":1}
I20260928 21:16:10.525744 20560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb4d345a5344a09b5aeb588b3f1967" candidate_uuid: "83ae8008f1d4447c81688aa37738efea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" is_pre_election: true
I20260928 21:16:10.526098 20560 raft_consensus.cc:2454] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ae8008f1d4447c81688aa37738efea in term 0.
I20260928 21:16:10.526081 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb4d345a5344a09b5aeb588b3f1967" candidate_uuid: "83ae8008f1d4447c81688aa37738efea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce13a5cd4c4945b8b7694b4505884f17" is_pre_election: true
I20260928 21:16:10.526372 20429 raft_consensus.cc:2454] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83ae8008f1d4447c81688aa37738efea in term 0.
I20260928 21:16:10.526607 20231 leader_election.cc:302] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f7bfee14c544bc8b37adbcd1cfc3cda, 83ae8008f1d4447c81688aa37738efea; no voters:
I20260928 21:16:10.526911 20631 raft_consensus.cc:2781] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260928 21:16:10.526994 20631 raft_consensus.cc:482] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260928 21:16:10.527048 20631 raft_consensus.cc:3037] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:10.528079 20631 raft_consensus.cc:504] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.528419 20631 leader_election.cc:288] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [CANDIDATE]: Term 1 election: Requested vote from peers 7f7bfee14c544bc8b37adbcd1cfc3cda (127.0.98.3:43821), ce13a5cd4c4945b8b7694b4505884f17 (127.0.98.2:41943)
I20260928 21:16:10.528946 20429 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb4d345a5344a09b5aeb588b3f1967" candidate_uuid: "83ae8008f1d4447c81688aa37738efea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce13a5cd4c4945b8b7694b4505884f17"
I20260928 21:16:10.529055 20429 raft_consensus.cc:3037] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:10.529067 20560 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f2cb4d345a5344a09b5aeb588b3f1967" candidate_uuid: "83ae8008f1d4447c81688aa37738efea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda"
I20260928 21:16:10.529227 20560 raft_consensus.cc:3037] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:10.530153 20429 raft_consensus.cc:2454] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ae8008f1d4447c81688aa37738efea in term 1.
I20260928 21:16:10.530414 20560 raft_consensus.cc:2454] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83ae8008f1d4447c81688aa37738efea in term 1.
I20260928 21:16:10.530443 20231 leader_election.cc:302] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83ae8008f1d4447c81688aa37738efea, ce13a5cd4c4945b8b7694b4505884f17; no voters:
I20260928 21:16:10.530632 20631 raft_consensus.cc:2781] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 1 FOLLOWER]: Leader election won for term 1
W20260928 21:16:10.530725 20476 tablet.cc:2369] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:10.530894 20631 raft_consensus.cc:686] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [term 1 LEADER]: Becoming Leader. State: Replica: 83ae8008f1d4447c81688aa37738efea, State: Running, Role: LEADER
I20260928 21:16:10.531221 20631 consensus_queue.cc:237] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } }
I20260928 21:16:10.534509 20156 catalog_manager.cc:5495] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea reported cstate change: term changed from 0 to 1, leader changed from <none> to 83ae8008f1d4447c81688aa37738efea (127.0.98.1). New cstate: current_term: 1 leader_uuid: "83ae8008f1d4447c81688aa37738efea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83ae8008f1d4447c81688aa37738efea" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 } health_report { overall_health: UNKNOWN } } }
I20260928 21:16:10.569039 20620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.565047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33464 (local address 127.0.98.1:43377)
0928 21:16:10.565176 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:10.565180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:10.565198 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:16:10.565376 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:10.565379 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:10.565444 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:10.565554 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:10.565764 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.566533 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.567358 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:10.567673 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:10.567778 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:10.568117 (+ 339us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:10.568602 (+ 485us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:10.568712 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:10.568811 (+ 99us) server_negotiation.cc:299] Negotiation successful
0928 21:16:10.568895 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260928 21:16:10.575544 20560 raft_consensus.cc:1261] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 1 FOLLOWER]: Refusing update from remote peer 83ae8008f1d4447c81688aa37738efea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:10.575544 20429 raft_consensus.cc:1261] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 1 FOLLOWER]: Refusing update from remote peer 83ae8008f1d4447c81688aa37738efea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:10.576315 20631 consensus_queue.cc:1035] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce13a5cd4c4945b8b7694b4505884f17" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:10.576718 20634 consensus_queue.cc:1035] T f2cb4d345a5344a09b5aeb588b3f1967 P 83ae8008f1d4447c81688aa37738efea [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f7bfee14c544bc8b37adbcd1cfc3cda" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:10.585484 20639 mvcc.cc:205] Tried to move back new op lower bound from 7334421178666991616 to 7334421178496221184. Current Snapshot: MvccSnapshot[applied={T|T < 7334421178666991616}]
I20260928 21:16:10.594760 20156 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:10.594966 20156 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50280:
table { table_name: "test-table" } modify_external_catalogs: true
I20260928 21:16:10.597878 20156 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6a82de4377b246a49491aa81c6960f3b: Sending DeleteTablet for 3 replicas of tablet f2cb4d345a5344a09b5aeb588b3f1967
I20260928 21:16:10.598928 20278 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cb4d345a5344a09b5aeb588b3f1967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:10 UTC) from {username='slave'} at 127.0.0.1:33462
I20260928 21:16:10.598937 20540 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cb4d345a5344a09b5aeb588b3f1967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:10 UTC) from {username='slave'} at 127.0.0.1:37648
I20260928 21:16:10.599066 20409 tablet_service.cc:1514] Processing DeleteTablet for tablet f2cb4d345a5344a09b5aeb588b3f1967 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-28 21:16:10 UTC) from {username='slave'} at 127.0.0.1:40162
I20260928 21:16:10.599543 20646 tablet_replica.cc:330] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: stopping tablet replica
I20260928 21:16:10.599704 20648 tablet_replica.cc:330] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: stopping tablet replica
I20260928 21:16:10.599923 20646 raft_consensus.cc:2229] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:10.600042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20216
I20260928 21:16:10.600704 20648 raft_consensus.cc:2229] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260928 21:16:10.601030 20646 raft_consensus.cc:2258] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:10.601264 20648 raft_consensus.cc:2258] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260928 21:16:10.602996 20646 ts_tablet_manager.cc:1905] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:10.602996 20648 ts_tablet_manager.cc:1905] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260928 21:16:10.605890 20648 ts_tablet_manager.cc:1918] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:10.605890 20646 ts_tablet_manager.cc:1918] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260928 21:16:10.606003 20648 log.cc:1196] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-1/wal/wals/f2cb4d345a5344a09b5aeb588b3f1967
I20260928 21:16:10.606003 20646 log.cc:1196] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Deleting WAL directory at /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790629916120794-392-0/minicluster-data/ts-2/wal/wals/f2cb4d345a5344a09b5aeb588b3f1967
I20260928 21:16:10.606338 20648 ts_tablet_manager.cc:1939] T f2cb4d345a5344a09b5aeb588b3f1967 P ce13a5cd4c4945b8b7694b4505884f17: Deleting consensus metadata
I20260928 21:16:10.606335 20646 ts_tablet_manager.cc:1939] T f2cb4d345a5344a09b5aeb588b3f1967 P 7f7bfee14c544bc8b37adbcd1cfc3cda: Deleting consensus metadata
I20260928 21:16:10.607890 20140 catalog_manager.cc:4841] TS 7f7bfee14c544bc8b37adbcd1cfc3cda (127.0.98.3:43821): tablet f2cb4d345a5344a09b5aeb588b3f1967 (table test-table [id=0d0826b0ef854528ba5928f17b96c05e]) successfully deleted
I20260928 21:16:10.608074 20140 catalog_manager.cc:4841] TS ce13a5cd4c4945b8b7694b4505884f17 (127.0.98.2:41943): tablet f2cb4d345a5344a09b5aeb588b3f1967 (table test-table [id=0d0826b0ef854528ba5928f17b96c05e]) successfully deleted
I20260928 21:16:10.608284 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20347
W20260928 21:16:10.608341 20140 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43377 (error 108)
I20260928 21:16:10.610270 20619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:10.609892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43377 (local address 127.0.0.1:33476)
0928 21:16:10.610068 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:10.610158 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260928 21:16:10.610459 20140 catalog_manager.cc:4584] TS 83ae8008f1d4447c81688aa37738efea (127.0.98.1:43377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f2cb4d345a5344a09b5aeb588b3f1967: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43377: connect: Connection refused (error 111)
I20260928 21:16:10.615008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20478
I20260928 21:16:10.620743 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20125
2026-09-28T21:16:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1260 ms)
[----------] 6 tests from EncryptionPolicyTest (10670 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:10 dist-test-slave-v7xz krb5kdc[20652](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:10 dist-test-slave-v7xz krb5kdc[20652](info): set up 2 sockets
Sep 28 21:16:10 dist-test-slave-v7xz krb5kdc[20652](info): commencing operation
krb5kdc: starting...
W20260928 21:16:11.664914 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:11 dist-test-slave-v7xz krb5kdc[20652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:11.787914 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:11.900763 20665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:11.901077 20665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:11.901121 20665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:11.905418 20665 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:11.905490 20665 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:11.905511 20665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:11.905529 20665 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:11.905580 20665 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:11.911234 20665 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39561
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20665
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:11.912402 20665 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:11.913615 20665 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:11.922209 20670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:11.922224 20671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:11.922231 20673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:11.922453 20665 server_base.cc:1034] running on GCE node
I20260928 21:16:11.923187 20665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:11.924469 20665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:11.925634 20665 hybrid_clock.cc:648] HybridClock initialized: now 1790630171925591 us; error 58 us; skew 500 ppm
Sep 28 21:16:11 dist-test-slave-v7xz krb5kdc[20652](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:11.928480 20665 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:11.929518 20665 webserver.cc:460] Webserver started at http://127.0.0.1:40841/ using document root <none> and password file <none>
I20260928 21:16:11.930104 20665 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:11.930215 20665 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:11.930469 20665 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:11.932569 20665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "2056f8593d7d4c81b75a72d0d5e99cca"
format_stamp: "Formatted at 2026-09-28 21:16:11 on dist-test-slave-v7xz"
server_key: "573909c88e8806b77c6df605e2e29a4e"
server_key_iv: "47e65b3150ce5eb60ce3235306d3a338"
server_key_version: "encryptionkey@0"
I20260928 21:16:11.933202 20665 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "2056f8593d7d4c81b75a72d0d5e99cca"
format_stamp: "Formatted at 2026-09-28 21:16:11 on dist-test-slave-v7xz"
server_key: "573909c88e8806b77c6df605e2e29a4e"
server_key_iv: "47e65b3150ce5eb60ce3235306d3a338"
server_key_version: "encryptionkey@0"
I20260928 21:16:11.936978 20665 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:16:11.939494 20680 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:11.940725 20665 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:11.940896 20665 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "2056f8593d7d4c81b75a72d0d5e99cca"
format_stamp: "Formatted at 2026-09-28 21:16:11 on dist-test-slave-v7xz"
server_key: "573909c88e8806b77c6df605e2e29a4e"
server_key_iv: "47e65b3150ce5eb60ce3235306d3a338"
server_key_version: "encryptionkey@0"
I20260928 21:16:11.941022 20665 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:11.958014 20665 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:11.960883 20665 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:11.961097 20665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:11.968637 20665 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39561
I20260928 21:16:11.968664 20732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39561 every 8 connection(s)
I20260928 21:16:11.969719 20665 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:11.972393 20733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:11.974491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20665
I20260928 21:16:11.974653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:11.974927 392 external_mini_cluster.cc:1428] Setting key 7d1323e2a4a22c9d5647dc2fc8c8b064
I20260928 21:16:11.978750 20733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca: Bootstrap starting.
I20260928 21:16:11.981317 20733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:11.982183 20733 log.cc:824] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:11.984206 20733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca: No bootstrap required, opened a new log
Sep 28 21:16:11 dist-test-slave-v7xz krb5kdc[20652](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:16:11.988620 20733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260928 21:16:11.988940 20733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:11.989001 20733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2056f8593d7d4c81b75a72d0d5e99cca, State: Initialized, Role: FOLLOWER
I20260928 21:16:11.989459 20733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260928 21:16:11.989584 20733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:11.989689 20733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:11.989830 20733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:11.990320 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:11.976380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41757 (local address 127.0.0.1:39561)
0928 21:16:11.976805 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:11.976844 (+ 39us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:11.976889 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:16:11.977699 (+ 810us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:11.977723 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:11.978155 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:11.978507 (+ 352us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:11.979247 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:11.980689 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:11.981641 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:11.982332 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:11.982499 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:11.985278 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:11.985283 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:11.985295 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:11.985320 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:11.986974 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:11.987529 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:11.987534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:11.987538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:11.987645 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:11.987979 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:11.987981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:11.987982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:11.988411 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:11.988626 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:11.989052 (+ 426us) server_negotiation.cc:299] Negotiation successful
0928 21:16:11.989326 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":113,"threads_started":1}
I20260928 21:16:11.990813 20733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260928 21:16:11.991168 20733 leader_election.cc:302] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2056f8593d7d4c81b75a72d0d5e99cca; no voters:
I20260928 21:16:11.991518 20733 leader_election.cc:288] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:11.991634 20738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:11.991888 20738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [term 1 LEADER]: Becoming Leader. State: Replica: 2056f8593d7d4c81b75a72d0d5e99cca, State: Running, Role: LEADER
I20260928 21:16:11.992194 20738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } }
I20260928 21:16:11.992753 20733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:11.993688 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:11.993954 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2056f8593d7d4c81b75a72d0d5e99cca. Latest consensus state: current_term: 1 leader_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } } }
I20260928 21:16:11.994087 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:11.994426 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2056f8593d7d4c81b75a72d0d5e99cca" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39561 } } }
I20260928 21:16:11.994527 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:11.994864 20747 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:11.998355 20747 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:12.004346 20747 catalog_manager.cc:1347] Generated new cluster ID: 3c3701a307ed4f62b3461b7c557835fd
I20260928 21:16:12.004411 20747 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:12.006126 20736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:11.997298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36955 (local address 127.0.0.1:39561)
0928 21:16:11.997409 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:11.997413 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:11.997670 (+ 257us) server_negotiation.cc:407] Connection header received
0928 21:16:11.997788 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:11.997791 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:11.997841 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:11.997927 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:11.998757 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:11.999554 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:12.000511 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:12.000761 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:12.000905 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:12.002382 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:12.002386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:12.002389 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:12.002414 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:12.004139 (+ 1725us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:12.004696 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:12.004700 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:12.004702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:12.004787 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:12.005175 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:12.005179 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:12.005181 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:12.005394 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:12.005553 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:12.005864 (+ 311us) server_negotiation.cc:299] Negotiation successful
0928 21:16:12.005967 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:16:12.038159 20747 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:12.039120 20747 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:12.048496 20747 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2056f8593d7d4c81b75a72d0d5e99cca: Generated new TSK 0
I20260928 21:16:12.049273 20747 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:12.062754 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20665
W20260928 21:16:12.068229 20650 connection.cc:535] client connection to 127.0.0.1:39561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39561): Connection reset by peer (error 104)
2026-09-28T21:16:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:12 dist-test-slave-v7xz krb5kdc[20764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:12 dist-test-slave-v7xz krb5kdc[20764](info): set up 2 sockets
Sep 28 21:16:12 dist-test-slave-v7xz krb5kdc[20764](info): commencing operation
krb5kdc: starting...
W20260928 21:16:13.101032 392 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:13 dist-test-slave-v7xz krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:13.223881 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:13.336777 20777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:13.337127 20777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:13.337211 20777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:13.341442 20777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:13.341529 20777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:13.341568 20777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:13.341624 20777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:13.341665 20777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:13.347153 20777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:13.348304 20777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:13.349486 20777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:13.358312 20783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:13.358466 20782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:13.358546 20777 server_base.cc:1034] running on GCE node
W20260928 21:16:13.358466 20785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:13.359280 20777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:13.360487 20777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:13.361680 20777 hybrid_clock.cc:648] HybridClock initialized: now 1790630173361631 us; error 74 us; skew 500 ppm
Sep 28 21:16:13 dist-test-slave-v7xz krb5kdc[20764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630173, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:13.364476 20777 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:13.365484 20777 webserver.cc:460] Webserver started at http://127.0.0.1:34629/ using document root <none> and password file <none>
I20260928 21:16:13.366061 20777 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:13.366154 20777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:13.366389 20777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:13.368470 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "a2d52ad516074e5dabc1aa2c2f835609"
format_stamp: "Formatted at 2026-09-28 21:16:13 on dist-test-slave-v7xz"
server_key: "6814d8b41be83570b20af0c1eca68d85"
server_key_iv: "2b1866262e3fa2882060d2e49ab7be4e"
server_key_version: "encryptionkey@0"
I20260928 21:16:13.369112 20777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "a2d52ad516074e5dabc1aa2c2f835609"
format_stamp: "Formatted at 2026-09-28 21:16:13 on dist-test-slave-v7xz"
server_key: "6814d8b41be83570b20af0c1eca68d85"
server_key_iv: "2b1866262e3fa2882060d2e49ab7be4e"
server_key_version: "encryptionkey@0"
I20260928 21:16:13.372980 20777 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260928 21:16:13.375406 20792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:13.376714 20777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:13.376909 20777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "a2d52ad516074e5dabc1aa2c2f835609"
format_stamp: "Formatted at 2026-09-28 21:16:13 on dist-test-slave-v7xz"
server_key: "6814d8b41be83570b20af0c1eca68d85"
server_key_iv: "2b1866262e3fa2882060d2e49ab7be4e"
server_key_version: "encryptionkey@0"
I20260928 21:16:13.377033 20777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:13.387353 20777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:13.390376 20777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:13.390585 20777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:13.397996 20777 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35041
I20260928 21:16:13.398022 20844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35041 every 8 connection(s)
I20260928 21:16:13.398969 20777 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:13.400300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20777
I20260928 21:16:13.400455 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:13.400753 392 external_mini_cluster.cc:1428] Setting key 423ef29e31c21f5a9820daebc68ca7af
I20260928 21:16:13.402202 20845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:13.408681 20845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609: Bootstrap starting.
I20260928 21:16:13.409816 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:13.402070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47899 (local address 127.0.0.1:35041)
0928 21:16:13.402489 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:13.402505 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:13.402541 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:16:13.403196 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:13.403213 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:13.403563 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:13.403883 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:13.404212 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:13.405311 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:13.406501 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:13.406998 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:13.407118 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:13.408088 (+ 970us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:13.408093 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:13.408106 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:13.408509 (+ 403us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:13.408567 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:13.408590 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:13.408692 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:13.408980 (+ 288us) server_negotiation.cc:299] Negotiation successful
0928 21:16:13.409098 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":154,"threads_started":1}
I20260928 21:16:13.411159 20845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:13.412046 20845 log.cc:824] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:13.413138 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:13.414341 20845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609: No bootstrap required, opened a new log
I20260928 21:16:13.417986 20845 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d52ad516074e5dabc1aa2c2f835609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 21:16:13.418313 20845 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:13.418406 20845 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2d52ad516074e5dabc1aa2c2f835609, State: Initialized, Role: FOLLOWER
I20260928 21:16:13.418898 20845 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d52ad516074e5dabc1aa2c2f835609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 21:16:13.419078 20845 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:13.419158 20845 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:13.419283 20845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:13.420240 20848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:13.415196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45235 (local address 127.0.0.1:35041)
0928 21:16:13.415329 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:13.415334 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:13.415511 (+ 177us) server_negotiation.cc:407] Connection header received
0928 21:16:13.415666 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:13.415668 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:13.415706 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:13.415780 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:13.416086 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:13.417120 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:13.418258 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:13.418644 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:13.418753 (+ 109us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:13.419752 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:13.419755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:13.419758 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:13.419771 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:13.419797 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:13.419807 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:13.419881 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:13.420023 (+ 142us) server_negotiation.cc:299] Negotiation successful
0928 21:16:13.420064 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:16:13.420573 20845 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d52ad516074e5dabc1aa2c2f835609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 21:16:13.420998 20845 leader_election.cc:302] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2d52ad516074e5dabc1aa2c2f835609; no voters:
I20260928 21:16:13.421350 20845 leader_election.cc:288] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:13.421442 20855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:13.421689 20855 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [term 1 LEADER]: Becoming Leader. State: Replica: a2d52ad516074e5dabc1aa2c2f835609, State: Running, Role: LEADER
I20260928 21:16:13.422056 20855 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d52ad516074e5dabc1aa2c2f835609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } }
I20260928 21:16:13.422607 20845 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:13.423695 20856 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2d52ad516074e5dabc1aa2c2f835609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d52ad516074e5dabc1aa2c2f835609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } } }
I20260928 21:16:13.423840 20856 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:13.424125 20857 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2d52ad516074e5dabc1aa2c2f835609. Latest consensus state: current_term: 1 leader_uuid: "a2d52ad516074e5dabc1aa2c2f835609" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2d52ad516074e5dabc1aa2c2f835609" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35041 } } }
I20260928 21:16:13.424232 20857 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:13.424324 20864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:13.427630 20864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:13.433604 20864 catalog_manager.cc:1347] Generated new cluster ID: c5ced814a8454de1a2bb0c67c9ab90c5
I20260928 21:16:13.433697 20864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:13.458163 20864 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:13.459215 20864 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:13.467664 20864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2d52ad516074e5dabc1aa2c2f835609: Generated new TSK 0
I20260928 21:16:13.468398 20864 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:13.472509 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20777
W20260928 21:16:13.477958 20762 connection.cc:535] client connection to 127.0.0.1:35041 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35041): Connection reset by peer (error 104)
2026-09-28T21:16:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1409 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:13 dist-test-slave-v7xz krb5kdc[20876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:13 dist-test-slave-v7xz krb5kdc[20876](info): set up 2 sockets
Sep 28 21:16:13 dist-test-slave-v7xz krb5kdc[20876](info): commencing operation
krb5kdc: starting...
W20260928 21:16:14.517154 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:14 dist-test-slave-v7xz krb5kdc[20876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630174, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:14.646085 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45233
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:14.761905 20889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:14.762212 20889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:14.762256 20889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:14.766335 20889 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:14.766412 20889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:14.766433 20889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:14.766450 20889 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:14.766466 20889 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:14.771986 20889 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45233
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:14.773078 20889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:14.774055 20889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:14.782585 20895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:14.782611 20897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:14.782692 20889 server_base.cc:1034] running on GCE node
W20260928 21:16:14.782603 20894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:14.783428 20889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:14.784578 20889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:14.785734 20889 hybrid_clock.cc:648] HybridClock initialized: now 1790630174785695 us; error 57 us; skew 500 ppm
Sep 28 21:16:14 dist-test-slave-v7xz krb5kdc[20876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:14.788427 20889 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:14.789422 20889 webserver.cc:460] Webserver started at http://127.0.0.1:40635/ using document root <none> and password file <none>
I20260928 21:16:14.790002 20889 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:14.790107 20889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:14.790349 20889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:14.792428 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb"
format_stamp: "Formatted at 2026-09-28 21:16:14 on dist-test-slave-v7xz"
server_key: "4820269a8c01abc6bf3df93a3ba36420"
server_key_iv: "34893c579a8cdf86df7f7bcd4da60c29"
server_key_version: "encryptionkey@0"
I20260928 21:16:14.793062 20889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb"
format_stamp: "Formatted at 2026-09-28 21:16:14 on dist-test-slave-v7xz"
server_key: "4820269a8c01abc6bf3df93a3ba36420"
server_key_iv: "34893c579a8cdf86df7f7bcd4da60c29"
server_key_version: "encryptionkey@0"
I20260928 21:16:14.796980 20889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:16:14.799347 20904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:14.800513 20889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:14.800652 20889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb"
format_stamp: "Formatted at 2026-09-28 21:16:14 on dist-test-slave-v7xz"
server_key: "4820269a8c01abc6bf3df93a3ba36420"
server_key_iv: "34893c579a8cdf86df7f7bcd4da60c29"
server_key_version: "encryptionkey@0"
I20260928 21:16:14.800767 20889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:14.805403 20889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:14.808254 20889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:14.808457 20889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:14.815670 20889 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45233
I20260928 21:16:14.815692 20956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45233 every 8 connection(s)
I20260928 21:16:14.816617 20889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:14.819293 20957 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:14.822705 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 20889
I20260928 21:16:14.822845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:14.823112 392 external_mini_cluster.cc:1428] Setting key 620a0cb0a62b81ec9517d31011894e0a
I20260928 21:16:14.825106 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb: Bootstrap starting.
I20260928 21:16:14.827358 20957 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:14.828188 20957 log.cc:824] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:14.828369 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:14.824512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57615 (local address 127.0.0.1:45233)
0928 21:16:14.824996 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:14.825010 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:14.825045 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:16:14.825708 (+ 663us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:14.825725 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:14.826006 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:14.826346 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:14.827158 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:14.827162 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:14.827174 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:14.827184 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:14.827240 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:14.827260 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:14.827377 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:14.827478 (+ 101us) server_negotiation.cc:299] Negotiation successful
0928 21:16:14.827649 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":132,"threads_started":1}
I20260928 21:16:14.830176 20957 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb: No bootstrap required, opened a new log
I20260928 21:16:14.831180 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:14.834441 20957 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260928 21:16:14.834780 20957 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:14.834875 20957 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e40fa3d6fbd40e29a00777aff5e3ccb, State: Initialized, Role: FOLLOWER
I20260928 21:16:14.834906 20960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:14.833229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41503 (local address 127.0.0.1:45233)
0928 21:16:14.833349 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:14.833361 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:14.833507 (+ 146us) server_negotiation.cc:407] Connection header received
0928 21:16:14.833631 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:14.833633 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:14.833664 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:14.833730 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:14.834445 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:14.834447 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:14.834449 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:14.834459 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:14.834481 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:14.834491 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:14.834583 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:14.834675 (+ 92us) server_negotiation.cc:299] Negotiation successful
0928 21:16:14.834706 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:16:14.835415 20957 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260928 21:16:14.835563 20957 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:14.835652 20957 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:14.835755 20957 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:14.836787 20957 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260928 21:16:14.837143 20957 leader_election.cc:302] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e40fa3d6fbd40e29a00777aff5e3ccb; no voters:
I20260928 21:16:14.837407 20957 leader_election.cc:288] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:14.837518 20967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:14.837770 20967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [term 1 LEADER]: Becoming Leader. State: Replica: 9e40fa3d6fbd40e29a00777aff5e3ccb, State: Running, Role: LEADER
I20260928 21:16:14.838120 20967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } }
I20260928 21:16:14.838721 20957 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:14.840058 20969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e40fa3d6fbd40e29a00777aff5e3ccb. Latest consensus state: current_term: 1 leader_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } } }
I20260928 21:16:14.840184 20969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:14.840487 20968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e40fa3d6fbd40e29a00777aff5e3ccb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45233 } } }
I20260928 21:16:14.840653 20968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:14.841187 20977 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:14.843842 20977 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:14.849553 20977 catalog_manager.cc:1347] Generated new cluster ID: e6ec16f4f2c745878d60a4fc92ef0d7c
I20260928 21:16:14.849632 20977 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:14.861761 20977 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:14.862141 20977 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:14.873466 20977 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e40fa3d6fbd40e29a00777aff5e3ccb: Generated new TSK 0
I20260928 21:16:14.874358 20977 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:14.878969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 20889
2026-09-28T21:16:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:14 dist-test-slave-v7xz krb5kdc[20988](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:14 dist-test-slave-v7xz krb5kdc[20988](info): set up 2 sockets
Sep 28 21:16:14 dist-test-slave-v7xz krb5kdc[20988](info): commencing operation
krb5kdc: starting...
W20260928 21:16:15.909165 392 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:15 dist-test-slave-v7xz krb5kdc[20988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:16.029975 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34973
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:16.141063 21001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:16.141384 21001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:16.141477 21001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:16.145640 21001 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:16.145731 21001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:16.145768 21001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:16.145833 21001 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:16.145874 21001 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:16.151298 21001 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34973
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21001
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:16.152387 21001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:16.153462 21001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:16.161913 21007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:16.161913 21009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:16.161989 21001 server_base.cc:1034] running on GCE node
W20260928 21:16:16.161942 21006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:16.162725 21001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:16.163841 21001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:16.164996 21001 hybrid_clock.cc:648] HybridClock initialized: now 1790630176164966 us; error 44 us; skew 500 ppm
Sep 28 21:16:16 dist-test-slave-v7xz krb5kdc[20988](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:16.175846 21001 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:16.176949 21001 webserver.cc:460] Webserver started at http://127.0.0.1:45357/ using document root <none> and password file <none>
I20260928 21:16:16.177783 21001 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:16.177901 21001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:16.178158 21001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:16.180315 21001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "ce04bc26236f4668903b57938c6a8571"
format_stamp: "Formatted at 2026-09-28 21:16:16 on dist-test-slave-v7xz"
server_key: "284fe2282ee35af2244a4a6fb6ec01cd"
server_key_iv: "e61cbead1044e07b35e02b5ce31e4b57"
server_key_version: "encryptionkey@0"
I20260928 21:16:16.180940 21001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "ce04bc26236f4668903b57938c6a8571"
format_stamp: "Formatted at 2026-09-28 21:16:16 on dist-test-slave-v7xz"
server_key: "284fe2282ee35af2244a4a6fb6ec01cd"
server_key_iv: "e61cbead1044e07b35e02b5ce31e4b57"
server_key_version: "encryptionkey@0"
I20260928 21:16:16.184885 21001 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260928 21:16:16.187523 21016 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:16.188949 21001 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:16.189083 21001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "ce04bc26236f4668903b57938c6a8571"
format_stamp: "Formatted at 2026-09-28 21:16:16 on dist-test-slave-v7xz"
server_key: "284fe2282ee35af2244a4a6fb6ec01cd"
server_key_iv: "e61cbead1044e07b35e02b5ce31e4b57"
server_key_version: "encryptionkey@0"
I20260928 21:16:16.189199 21001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:16.203788 21001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:16.207864 21001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:16.208185 21001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:16.215373 21001 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34973
I20260928 21:16:16.215396 21068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34973 every 8 connection(s)
I20260928 21:16:16.216377 21001 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:16.219184 21069 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:16.224606 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571: Bootstrap starting.
I20260928 21:16:16.226606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21001
I20260928 21:16:16.226758 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:16.227020 392 external_mini_cluster.cc:1428] Setting key 0265c80204c970d80e6060459cc62be7
I20260928 21:16:16.227206 21069 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:16.228012 21069 log.cc:824] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:16.230026 21069 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571: No bootstrap required, opened a new log
I20260928 21:16:16.233498 21069 raft_consensus.cc:348] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04bc26236f4668903b57938c6a8571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34973 } }
I20260928 21:16:16.233731 21069 raft_consensus.cc:374] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:16.233810 21069 raft_consensus.cc:729] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce04bc26236f4668903b57938c6a8571, State: Initialized, Role: FOLLOWER
I20260928 21:16:16.234302 21069 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04bc26236f4668903b57938c6a8571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34973 } }
I20260928 21:16:16.234457 21069 raft_consensus.cc:388] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:16.234543 21069 raft_consensus.cc:482] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:16.234642 21069 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 0 FOLLOWER]: Advancing to term 1
Sep 28 21:16:16 dist-test-slave-v7xz krb5kdc[20988](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:16:16.235831 21069 raft_consensus.cc:504] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04bc26236f4668903b57938c6a8571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34973 } }
I20260928 21:16:16.236209 21069 leader_election.cc:302] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ce04bc26236f4668903b57938c6a8571; no voters:
I20260928 21:16:16.236580 21069 leader_election.cc:288] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:16.236698 21074 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:16.236989 21074 raft_consensus.cc:686] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [term 1 LEADER]: Becoming Leader. State: Replica: ce04bc26236f4668903b57938c6a8571, State: Running, Role: LEADER
I20260928 21:16:16.237394 21074 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04bc26236f4668903b57938c6a8571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34973 } }
I20260928 21:16:16.237859 21069 sys_catalog.cc:564] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:16.239138 21076 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce04bc26236f4668903b57938c6a8571. Latest consensus state: current_term: 1 leader_uuid: "ce04bc26236f4668903b57938c6a8571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04bc26236f4668903b57938c6a8571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34973 } } }
I20260928 21:16:16.239287 21076 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:16.239636 21075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ce04bc26236f4668903b57938c6a8571" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ce04bc26236f4668903b57938c6a8571" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34973 } } }
I20260928 21:16:16.239701 21084 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:16.239806 21075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:16.242704 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:16.228086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53215 (local address 127.0.0.1:34973)
0928 21:16:16.228657 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:16.228679 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:16.228724 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:16:16.229440 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:16.229457 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:16.229790 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:16.230136 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:16.230922 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:16.231885 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:16.232739 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:16.233295 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:16.233473 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:16.236394 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:16.236400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:16.236417 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:16.236451 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:16.238963 (+ 2512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:16.239553 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:16.239562 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:16.239571 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:16.239729 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:16.240037 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:16.240040 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:16.240042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:16.240535 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:16.240718 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:16.241517 (+ 799us) server_negotiation.cc:299] Negotiation successful
0928 21:16:16.241816 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":411,"thread_start_us":153,"threads_started":1}
I20260928 21:16:16.243184 21084 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:16.245621 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:16.249028 21084 catalog_manager.cc:1347] Generated new cluster ID: 439741c3f8a4464cb0d6b84f56d348f0
I20260928 21:16:16.249089 21084 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:16.257469 21072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:16.247813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42423 (local address 127.0.0.1:34973)
0928 21:16:16.247914 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:16.247918 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:16.248231 (+ 313us) server_negotiation.cc:407] Connection header received
0928 21:16:16.248355 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:16.248358 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:16.248406 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:16.248494 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:16.249387 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:16.250221 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:16.251410 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:16.251796 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:16.251876 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:16.253692 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:16.253696 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:16.253698 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:16.253726 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:16.255481 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:16.256000 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:16.256004 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:16.256006 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:16.256090 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:16.256391 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:16.256394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:16.256396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:16.256615 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:16.256718 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:16.257150 (+ 432us) server_negotiation.cc:299] Negotiation successful
0928 21:16:16.257281 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:16:16.260900 21084 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:16.261840 21084 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:16.266587 21084 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ce04bc26236f4668903b57938c6a8571: Generated new TSK 0
I20260928 21:16:16.267321 21084 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:16.272744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21001
2026-09-28T21:16:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:16 dist-test-slave-v7xz krb5kdc[21100](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:16 dist-test-slave-v7xz krb5kdc[21100](info): set up 2 sockets
Sep 28 21:16:16 dist-test-slave-v7xz krb5kdc[21100](info): commencing operation
krb5kdc: starting...
W20260928 21:16:17.309170 392 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:17 dist-test-slave-v7xz krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630177, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:17.427209 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43049
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:17.535565 21113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:17.535827 21113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:17.535868 21113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:17.539868 21113 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:17.539927 21113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:17.539945 21113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:17.539963 21113 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:17.539978 21113 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:17.545252 21113 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21113
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:17.546252 21113 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:17.547230 21113 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:17.555577 21118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:17.555595 21119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:17.555675 21113 server_base.cc:1034] running on GCE node
W20260928 21:16:17.555590 21121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:17.556438 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:17.557612 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:17.558779 21113 hybrid_clock.cc:648] HybridClock initialized: now 1790630177558762 us; error 37 us; skew 500 ppm
Sep 28 21:16:17 dist-test-slave-v7xz krb5kdc[21100](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630177, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:17.561674 21113 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:17.562655 21113 webserver.cc:460] Webserver started at http://127.0.0.1:36617/ using document root <none> and password file <none>
I20260928 21:16:17.563235 21113 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:17.563299 21113 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:17.563540 21113 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:17.565719 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "7a5a67f11c8143af82a7882515deb31a"
format_stamp: "Formatted at 2026-09-28 21:16:17 on dist-test-slave-v7xz"
server_key: "73b68bcc91c534a30cc2e8fb327a5528"
server_key_iv: "c33bd1dccf57cd5c10d8b6d7b3ceb415"
server_key_version: "encryptionkey@0"
I20260928 21:16:17.566306 21113 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "7a5a67f11c8143af82a7882515deb31a"
format_stamp: "Formatted at 2026-09-28 21:16:17 on dist-test-slave-v7xz"
server_key: "73b68bcc91c534a30cc2e8fb327a5528"
server_key_iv: "c33bd1dccf57cd5c10d8b6d7b3ceb415"
server_key_version: "encryptionkey@0"
I20260928 21:16:17.570182 21113 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260928 21:16:17.572842 21128 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:17.574169 21113 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:16:17.574316 21113 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "7a5a67f11c8143af82a7882515deb31a"
format_stamp: "Formatted at 2026-09-28 21:16:17 on dist-test-slave-v7xz"
server_key: "73b68bcc91c534a30cc2e8fb327a5528"
server_key_iv: "c33bd1dccf57cd5c10d8b6d7b3ceb415"
server_key_version: "encryptionkey@0"
I20260928 21:16:17.574441 21113 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:17.594401 21113 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:17.597615 21113 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:17.597846 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:17.605175 21113 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43049
I20260928 21:16:17.605198 21180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43049 every 8 connection(s)
I20260928 21:16:17.606125 21113 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:17.608752 21181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:17.613225 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21113
I20260928 21:16:17.613374 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:17.613621 392 external_mini_cluster.cc:1428] Setting key 599ca1e6bbef1e8926e8c2d118507f02
I20260928 21:16:17.614462 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a: Bootstrap starting.
I20260928 21:16:17.617131 21181 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:17.618039 21181 log.cc:824] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:17.620082 21181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a: No bootstrap required, opened a new log
I20260928 21:16:17.624221 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:17.615088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56907 (local address 127.0.0.1:43049)
0928 21:16:17.615559 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:17.615580 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:17.615629 (+ 49us) server_negotiation.cc:407] Connection header received
0928 21:16:17.616443 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:17.616467 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:17.616943 (+ 476us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:17.617332 (+ 389us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:17.617767 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:17.619100 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:17.620064 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:17.620751 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:17.620933 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:17.621965 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:17.621972 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:17.621989 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:17.622509 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:17.622575 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:17.622605 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:17.622704 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:17.622937 (+ 233us) server_negotiation.cc:299] Negotiation successful
0928 21:16:17.623221 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":127,"threads_started":1}
I20260928 21:16:17.624671 21181 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a67f11c8143af82a7882515deb31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260928 21:16:17.625000 21181 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:17.625074 21181 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a5a67f11c8143af82a7882515deb31a, State: Initialized, Role: FOLLOWER
I20260928 21:16:17.625595 21181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a67f11c8143af82a7882515deb31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260928 21:16:17.625756 21181 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:17.625810 21181 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:17.625905 21181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:17.627126 21181 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a67f11c8143af82a7882515deb31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260928 21:16:17.627489 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:17.627477 21181 leader_election.cc:302] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a5a67f11c8143af82a7882515deb31a; no voters:
I20260928 21:16:17.627796 21181 leader_election.cc:288] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:17.627930 21186 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:17.628177 21186 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [term 1 LEADER]: Becoming Leader. State: Replica: 7a5a67f11c8143af82a7882515deb31a, State: Running, Role: LEADER
I20260928 21:16:17.628582 21186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a67f11c8143af82a7882515deb31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } }
I20260928 21:16:17.629154 21181 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:17.630311 21187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a5a67f11c8143af82a7882515deb31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a67f11c8143af82a7882515deb31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } } }
I20260928 21:16:17.630445 21187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:17.630908 21200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:17.630903 21188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a5a67f11c8143af82a7882515deb31a. Latest consensus state: current_term: 1 leader_uuid: "7a5a67f11c8143af82a7882515deb31a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a5a67f11c8143af82a7882515deb31a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43049 } } }
I20260928 21:16:17.631126 21188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:17.634262 21200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:17.635861 21184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:17.629794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35241 (local address 127.0.0.1:43049)
0928 21:16:17.629911 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:17.629916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:17.631645 (+ 1729us) server_negotiation.cc:407] Connection header received
0928 21:16:17.631778 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:17.631782 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:17.631834 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:17.631925 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:17.632208 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:17.633018 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:17.633861 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:17.634222 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:17.634315 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:17.635158 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:17.635161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:17.635166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:17.635184 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:17.635219 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:17.635236 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:17.635306 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:17.635711 (+ 405us) server_negotiation.cc:299] Negotiation successful
0928 21:16:17.635760 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:16:17.640450 21200 catalog_manager.cc:1347] Generated new cluster ID: c445c29fdfa14031a47b40930e7ffd2f
I20260928 21:16:17.640538 21200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:17.667142 21200 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:17.668126 21200 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:17.677053 21200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a5a67f11c8143af82a7882515deb31a: Generated new TSK 0
I20260928 21:16:17.677858 21200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:17.684208 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21113
2026-09-28T21:16:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:17 dist-test-slave-v7xz krb5kdc[21212](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:17 dist-test-slave-v7xz krb5kdc[21212](info): set up 2 sockets
Sep 28 21:16:17 dist-test-slave-v7xz krb5kdc[21212](info): commencing operation
krb5kdc: starting...
W20260928 21:16:18.749176 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:18 dist-test-slave-v7xz krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:18.868263 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43461
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:18.978928 21225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:18.979259 21225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:18.979341 21225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:18.983517 21225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:18.983619 21225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:18.983659 21225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:18.983719 21225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:18.983764 21225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:18.989400 21225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43461
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21225
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:18.990542 21225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:18.991596 21225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:16:19.000103 21225 server_base.cc:1034] running on GCE node
W20260928 21:16:19.000056 21233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:19.000066 21231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:19.000082 21230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:19.001036 21225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:19.002144 21225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:19.003302 21225 hybrid_clock.cc:648] HybridClock initialized: now 1790630179003289 us; error 27 us; skew 500 ppm
Sep 28 21:16:19 dist-test-slave-v7xz krb5kdc[21212](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:19.006342 21225 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:19.007233 21225 webserver.cc:460] Webserver started at http://127.0.0.1:40905/ using document root <none> and password file <none>
I20260928 21:16:19.007752 21225 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:19.007809 21225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:19.007972 21225 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:19.010069 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "5dadeeb194d247d9948b8316c0faa829"
format_stamp: "Formatted at 2026-09-28 21:16:19 on dist-test-slave-v7xz"
server_key: "0907040024deeddc242cde781c4f774c"
server_key_iv: "9caf19e13d550cf694f874f86b18f453"
server_key_version: "encryptionkey@0"
I20260928 21:16:19.010584 21225 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "5dadeeb194d247d9948b8316c0faa829"
format_stamp: "Formatted at 2026-09-28 21:16:19 on dist-test-slave-v7xz"
server_key: "0907040024deeddc242cde781c4f774c"
server_key_iv: "9caf19e13d550cf694f874f86b18f453"
server_key_version: "encryptionkey@0"
I20260928 21:16:19.014313 21225 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:19.016595 21240 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:19.017867 21225 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:19.018009 21225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "5dadeeb194d247d9948b8316c0faa829"
format_stamp: "Formatted at 2026-09-28 21:16:19 on dist-test-slave-v7xz"
server_key: "0907040024deeddc242cde781c4f774c"
server_key_iv: "9caf19e13d550cf694f874f86b18f453"
server_key_version: "encryptionkey@0"
I20260928 21:16:19.018126 21225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:19.022753 21225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:19.026193 21225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:19.026399 21225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:19.034176 21225 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43461
I20260928 21:16:19.034210 21292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43461 every 8 connection(s)
I20260928 21:16:19.035203 21225 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:19.038163 21293 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:19.043933 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829: Bootstrap starting.
I20260928 21:16:19.044425 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21225
I20260928 21:16:19.044580 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:19.044914 392 external_mini_cluster.cc:1428] Setting key 232d2e2a0ef4c7f60e06f45236655d66
I20260928 21:16:19.046783 21293 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:19.047567 21293 log.cc:824] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:19.049978 21293 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829: No bootstrap required, opened a new log
I20260928 21:16:19.050457 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:19.046250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53877 (local address 127.0.0.1:43461)
0928 21:16:19.046687 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:19.046702 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:19.046745 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:16:19.047447 (+ 702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:19.047465 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:19.047815 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:19.048192 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:19.049139 (+ 947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:19.049144 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:19.049156 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:19.049166 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:19.049219 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:19.049239 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:19.049329 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:19.049474 (+ 145us) server_negotiation.cc:299] Negotiation successful
0928 21:16:19.049658 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":139,"threads_started":1}
I20260928 21:16:19.053440 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:19.054183 21293 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dadeeb194d247d9948b8316c0faa829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260928 21:16:19.054473 21293 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:19.054551 21293 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dadeeb194d247d9948b8316c0faa829, State: Initialized, Role: FOLLOWER
I20260928 21:16:19.055034 21293 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dadeeb194d247d9948b8316c0faa829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260928 21:16:19.055189 21293 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:19.055291 21293 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:19.055393 21293 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:19.056703 21293 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dadeeb194d247d9948b8316c0faa829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260928 21:16:19.057114 21293 leader_election.cc:302] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dadeeb194d247d9948b8316c0faa829; no voters:
I20260928 21:16:19.057202 21296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:19.055545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56365 (local address 127.0.0.1:43461)
0928 21:16:19.055663 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:19.055667 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:19.055883 (+ 216us) server_negotiation.cc:407] Connection header received
0928 21:16:19.055975 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:19.055978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:19.056014 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:19.056089 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:19.056763 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:19.056766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:19.056769 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:19.056782 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:19.056814 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:19.056863 (+ 49us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:19.056951 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:19.057016 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:16:19.057054 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:16:19.057503 21293 leader_election.cc:288] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:19.057622 21303 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:19.057842 21303 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [term 1 LEADER]: Becoming Leader. State: Replica: 5dadeeb194d247d9948b8316c0faa829, State: Running, Role: LEADER
I20260928 21:16:19.058408 21303 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dadeeb194d247d9948b8316c0faa829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } }
I20260928 21:16:19.058871 21293 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:19.060271 21305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dadeeb194d247d9948b8316c0faa829. Latest consensus state: current_term: 1 leader_uuid: "5dadeeb194d247d9948b8316c0faa829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dadeeb194d247d9948b8316c0faa829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } } }
I20260928 21:16:19.060287 21304 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dadeeb194d247d9948b8316c0faa829" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dadeeb194d247d9948b8316c0faa829" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43461 } } }
I20260928 21:16:19.060407 21305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:19.060417 21304 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:19.060972 21313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:19.064560 21313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:19.070940 21313 catalog_manager.cc:1347] Generated new cluster ID: 4e1d6bdc3ffb4485810977aa16a291ad
I20260928 21:16:19.071041 21313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:19.087541 21313 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:19.087911 21313 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:19.093027 21313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dadeeb194d247d9948b8316c0faa829: Generated new TSK 0
I20260928 21:16:19.093832 21313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:19.101300 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21225
2026-09-28T21:16:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:19 dist-test-slave-v7xz krb5kdc[21324](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:19 dist-test-slave-v7xz krb5kdc[21324](info): set up 2 sockets
Sep 28 21:16:19 dist-test-slave-v7xz krb5kdc[21324](info): commencing operation
krb5kdc: starting...
W20260928 21:16:20.137128 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:20 dist-test-slave-v7xz krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:20.259963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37191
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37191
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:20.371994 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:20.372306 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:20.372403 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:20.376451 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:20.376550 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:20.376590 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:20.376646 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:20.376685 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:20.382025 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37191
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:20.383142 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:20.384182 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:20.393210 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:20.393205 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:20.393209 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:20.393258 21337 server_base.cc:1034] running on GCE node
I20260928 21:16:20.394198 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:20.395398 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:20.396603 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790630180396590 us; error 26 us; skew 500 ppm
Sep 28 21:16:20 dist-test-slave-v7xz krb5kdc[21324](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630180, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:20.399614 21337 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:20.400621 21337 webserver.cc:460] Webserver started at http://127.0.0.1:43427/ using document root <none> and password file <none>
I20260928 21:16:20.401268 21337 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:20.401367 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:20.401607 21337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:20.403750 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "c2e3b3d2edd7459cb8f6803e828ad50d"
format_stamp: "Formatted at 2026-09-28 21:16:20 on dist-test-slave-v7xz"
server_key: "56dbc650a5299821e182c2ed15b9634d"
server_key_iv: "893000d0f753e88e96bcb02ea69e6f1b"
server_key_version: "encryptionkey@0"
I20260928 21:16:20.404331 21337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "c2e3b3d2edd7459cb8f6803e828ad50d"
format_stamp: "Formatted at 2026-09-28 21:16:20 on dist-test-slave-v7xz"
server_key: "56dbc650a5299821e182c2ed15b9634d"
server_key_iv: "893000d0f753e88e96bcb02ea69e6f1b"
server_key_version: "encryptionkey@0"
I20260928 21:16:20.408385 21337 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:16:20.411252 21352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:20.412607 21337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:20.412767 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "c2e3b3d2edd7459cb8f6803e828ad50d"
format_stamp: "Formatted at 2026-09-28 21:16:20 on dist-test-slave-v7xz"
server_key: "56dbc650a5299821e182c2ed15b9634d"
server_key_iv: "893000d0f753e88e96bcb02ea69e6f1b"
server_key_version: "encryptionkey@0"
I20260928 21:16:20.412963 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:20.437633 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:20.440593 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:20.440809 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:20.448076 21337 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37191
I20260928 21:16:20.448102 21404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37191 every 8 connection(s)
I20260928 21:16:20.449105 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:20.451902 21405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:20.456696 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21337
I20260928 21:16:20.456815 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:20.457141 392 external_mini_cluster.cc:1428] Setting key 7cf1ec7a8f03b20bcba8e8c73f934967
I20260928 21:16:20.457476 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d: Bootstrap starting.
I20260928 21:16:20.459941 21405 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:20.460667 21405 log.cc:824] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:20.462498 21405 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d: No bootstrap required, opened a new log
Sep 28 21:16:20 dist-test-slave-v7xz krb5kdc[21324](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:16:20.466271 21405 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260928 21:16:20.466598 21405 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:20.466710 21405 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2e3b3d2edd7459cb8f6803e828ad50d, State: Initialized, Role: FOLLOWER
I20260928 21:16:20.467181 21405 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260928 21:16:20.467337 21405 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:20.467443 21405 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:20.467581 21405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:20.468874 21405 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260928 21:16:20.469282 21405 leader_election.cc:302] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2e3b3d2edd7459cb8f6803e828ad50d; no voters:
I20260928 21:16:20.469617 21405 leader_election.cc:288] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:20.469822 21410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:20.470130 21410 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [term 1 LEADER]: Becoming Leader. State: Replica: c2e3b3d2edd7459cb8f6803e828ad50d, State: Running, Role: LEADER
I20260928 21:16:20.470504 21410 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } }
I20260928 21:16:20.470955 21405 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:20.471884 21410 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2e3b3d2edd7459cb8f6803e828ad50d. Latest consensus state: current_term: 1 leader_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } } }
I20260928 21:16:20.472545 21410 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:20.472920 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2e3b3d2edd7459cb8f6803e828ad50d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37191 } } }
I20260928 21:16:20.473148 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:20.458501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:44031 (local address 127.0.0.1:37191)
0928 21:16:20.458963 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:20.458977 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:20.459017 (+ 40us) server_negotiation.cc:407] Connection header received
0928 21:16:20.459649 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:20.459666 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:20.460011 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:20.460402 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:20.461248 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:20.462228 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:20.463164 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:20.463659 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:20.463833 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:20.466816 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:20.466824 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:20.466848 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:20.466878 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:20.469065 (+ 2187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:20.469839 (+ 774us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:20.469850 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:20.469858 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:20.470028 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:20.470417 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:20.470420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:20.470423 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:20.470986 (+ 563us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:20.471224 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:20.471796 (+ 572us) server_negotiation.cc:299] Negotiation successful
0928 21:16:20.472061 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":318,"thread_start_us":155,"threads_started":1}
I20260928 21:16:20.473234 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:20.476758 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260928 21:16:20.477687 21425 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:20.477777 21425 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:20.477852 21424 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:20.478746 21424 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:20.485464 21424 catalog_manager.cc:1347] Generated new cluster ID: b174e7c06cfa48749259d8d21f80292e
I20260928 21:16:20.485543 21424 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:20.490677 21408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:20.481279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:48701 (local address 127.0.0.1:37191)
0928 21:16:20.481403 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:20.481407 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:20.481746 (+ 339us) server_negotiation.cc:407] Connection header received
0928 21:16:20.481860 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:20.481863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:20.481915 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:20.482009 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:20.482880 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:20.483717 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:20.484567 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:20.484818 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:20.485000 (+ 182us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:20.487011 (+ 2011us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:20.487015 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:20.487018 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:20.487046 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:20.488756 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:20.489321 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:20.489325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:20.489327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:20.489421 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:20.489756 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:20.489759 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:20.489761 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:20.489993 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:20.490143 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:20.490399 (+ 256us) server_negotiation.cc:299] Negotiation successful
0928 21:16:20.490510 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:16:20.511873 21424 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:20.512938 21424 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:20.517282 21424 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2e3b3d2edd7459cb8f6803e828ad50d: Generated new TSK 0
I20260928 21:16:20.518158 21424 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:20.531711 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21337
W20260928 21:16:20.536561 21322 connection.cc:535] client connection to 127.0.0.1:37191 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37191): Connection reset by peer (error 104)
2026-09-28T21:16:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1429 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:20 dist-test-slave-v7xz krb5kdc[21436](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:20 dist-test-slave-v7xz krb5kdc[21436](info): set up 2 sockets
Sep 28 21:16:20 dist-test-slave-v7xz krb5kdc[21436](info): commencing operation
krb5kdc: starting...
W20260928 21:16:21.581054 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:21 dist-test-slave-v7xz krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:21.696758 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37017
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37017
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:21.806962 21449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:21.807312 21449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:21.807403 21449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:21.811507 21449 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:21.811611 21449 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:21.811652 21449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:21.811689 21449 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:21.811721 21449 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:21.817243 21449 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21449
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:21.818368 21449 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:21.819398 21449 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:21.827939 21454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:21.828003 21449 server_base.cc:1034] running on GCE node
W20260928 21:16:21.828022 21457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:21.827946 21455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:21.828775 21449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:21.829874 21449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:21.831027 21449 hybrid_clock.cc:648] HybridClock initialized: now 1790630181831010 us; error 29 us; skew 500 ppm
Sep 28 21:16:21 dist-test-slave-v7xz krb5kdc[21436](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:21.837673 21449 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:21.838562 21449 webserver.cc:460] Webserver started at http://127.0.0.1:40365/ using document root <none> and password file <none>
I20260928 21:16:21.839074 21449 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:21.839131 21449 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:21.839291 21449 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:21.841300 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1"
format_stamp: "Formatted at 2026-09-28 21:16:21 on dist-test-slave-v7xz"
server_key: "b8b5b89263f402f78abf33bdd3176c65"
server_key_iv: "19608fcd1bb7c4cd3822177fcf85006b"
server_key_version: "encryptionkey@0"
I20260928 21:16:21.841801 21449 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1"
format_stamp: "Formatted at 2026-09-28 21:16:21 on dist-test-slave-v7xz"
server_key: "b8b5b89263f402f78abf33bdd3176c65"
server_key_iv: "19608fcd1bb7c4cd3822177fcf85006b"
server_key_version: "encryptionkey@0"
I20260928 21:16:21.845325 21449 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:16:21.847515 21464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:21.848697 21449 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:21.848804 21449 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1"
format_stamp: "Formatted at 2026-09-28 21:16:21 on dist-test-slave-v7xz"
server_key: "b8b5b89263f402f78abf33bdd3176c65"
server_key_iv: "19608fcd1bb7c4cd3822177fcf85006b"
server_key_version: "encryptionkey@0"
I20260928 21:16:21.848956 21449 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:21.861502 21449 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:21.864122 21449 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:21.864269 21449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:21.871418 21449 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37017
I20260928 21:16:21.871445 21516 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37017 every 8 connection(s)
I20260928 21:16:21.872363 21449 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:21.872617 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21449
I20260928 21:16:21.872773 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:21.873117 392 external_mini_cluster.cc:1428] Setting key 929f92b849de28dda0951997f93d464f
I20260928 21:16:21.875676 21517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:21.882483 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1: Bootstrap starting.
I20260928 21:16:21.883366 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:21.874543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:36325 (local address 127.0.0.1:37017)
0928 21:16:21.874963 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:21.874981 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:21.875025 (+ 44us) server_negotiation.cc:407] Connection header received
0928 21:16:21.875886 (+ 861us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:21.875909 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:21.876334 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:21.876696 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:21.877140 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:21.878629 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:21.879581 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:21.880362 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:21.880533 (+ 171us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:21.881451 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:21.881460 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:21.881478 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:21.881971 (+ 493us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:21.882039 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:21.882069 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:21.882199 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:21.882350 (+ 151us) server_negotiation.cc:299] Negotiation successful
0928 21:16:21.882480 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":117,"threads_started":1}
I20260928 21:16:21.885120 21517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:21.886010 21517 log.cc:824] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:21.886454 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:21.888057 21517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1: No bootstrap required, opened a new log
I20260928 21:16:21.891206 21517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37017 } }
I20260928 21:16:21.891480 21517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:21.891603 21517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7fe76e9b3a50432cb02b9b6eb278b6f1, State: Initialized, Role: FOLLOWER
I20260928 21:16:21.892125 21517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37017 } }
I20260928 21:16:21.892310 21517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:21.892398 21517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:21.892518 21517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:21.893124 21520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:21.888529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:54607 (local address 127.0.0.1:37017)
0928 21:16:21.888703 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:21.888707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:21.888821 (+ 114us) server_negotiation.cc:407] Connection header received
0928 21:16:21.889015 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:21.889019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:21.889065 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:21.889163 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:21.889434 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:21.890379 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:21.891203 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:21.891462 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:21.891589 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:21.892525 (+ 936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:21.892529 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:21.892531 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:21.892569 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:21.892616 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:21.892631 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:21.892719 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:21.892920 (+ 201us) server_negotiation.cc:299] Negotiation successful
0928 21:16:21.892964 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 21:16:21.893698 21517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37017 } }
I20260928 21:16:21.894053 21517 leader_election.cc:302] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7fe76e9b3a50432cb02b9b6eb278b6f1; no voters:
I20260928 21:16:21.894366 21517 leader_election.cc:288] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:21.894483 21527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:21.894726 21527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [term 1 LEADER]: Becoming Leader. State: Replica: 7fe76e9b3a50432cb02b9b6eb278b6f1, State: Running, Role: LEADER
I20260928 21:16:21.895134 21527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37017 } }
I20260928 21:16:21.895685 21517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:21.896718 21529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7fe76e9b3a50432cb02b9b6eb278b6f1. Latest consensus state: current_term: 1 leader_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37017 } } }
I20260928 21:16:21.896880 21529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:21.897306 21536 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:21.897321 21528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7fe76e9b3a50432cb02b9b6eb278b6f1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37017 } } }
I20260928 21:16:21.897492 21528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:21.900223 21536 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:21.906183 21536 catalog_manager.cc:1347] Generated new cluster ID: 8278a54894c6414198b87707a5f8a716
I20260928 21:16:21.906268 21536 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:21.927799 21536 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:21.928913 21536 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:21.938217 21536 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7fe76e9b3a50432cb02b9b6eb278b6f1: Generated new TSK 0
I20260928 21:16:21.939003 21536 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:21.952890 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21449
W20260928 21:16:21.958412 21434 connection.cc:535] client connection to 127.0.0.1:37017 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37017): Connection reset by peer (error 104)
2026-09-28T21:16:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1421 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:22 dist-test-slave-v7xz krb5kdc[21548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:22 dist-test-slave-v7xz krb5kdc[21548](info): set up 2 sockets
Sep 28 21:16:22 dist-test-slave-v7xz krb5kdc[21548](info): commencing operation
krb5kdc: starting...
W20260928 21:16:22.985096 392 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:23 dist-test-slave-v7xz krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:23.107856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42463
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33749
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:23.216584 21561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:23.216903 21561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:23.217018 21561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:23.220956 21561 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:23.221050 21561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:23.221094 21561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:23.221130 21561 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:23.221166 21561 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:23.226258 21561 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33749
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:23.227252 21561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:23.228230 21561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:23.237181 21566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:23.237155 21567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:23.237176 21569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:23.237632 21561 server_base.cc:1034] running on GCE node
I20260928 21:16:23.238164 21561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:23.239239 21561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:23.240434 21561 hybrid_clock.cc:648] HybridClock initialized: now 1790630183240422 us; error 27 us; skew 500 ppm
Sep 28 21:16:23 dist-test-slave-v7xz krb5kdc[21548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:23.243121 21561 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:23.243943 21561 webserver.cc:460] Webserver started at http://127.0.0.1:46293/ using document root <none> and password file <none>
I20260928 21:16:23.244474 21561 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:23.244540 21561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:23.244702 21561 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:23.246757 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "795322883b3749c48e03f3c06178b79c"
format_stamp: "Formatted at 2026-09-28 21:16:23 on dist-test-slave-v7xz"
server_key: "c640031d563a8b3eb5124aa0008ad51a"
server_key_iv: "0680b05ac6f3b4d9be11fd3b1bde7428"
server_key_version: "encryptionkey@0"
I20260928 21:16:23.247254 21561 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "795322883b3749c48e03f3c06178b79c"
format_stamp: "Formatted at 2026-09-28 21:16:23 on dist-test-slave-v7xz"
server_key: "c640031d563a8b3eb5124aa0008ad51a"
server_key_iv: "0680b05ac6f3b4d9be11fd3b1bde7428"
server_key_version: "encryptionkey@0"
I20260928 21:16:23.250851 21561 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260928 21:16:23.253153 21576 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:23.254385 21561 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260928 21:16:23.254487 21561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "795322883b3749c48e03f3c06178b79c"
format_stamp: "Formatted at 2026-09-28 21:16:23 on dist-test-slave-v7xz"
server_key: "c640031d563a8b3eb5124aa0008ad51a"
server_key_iv: "0680b05ac6f3b4d9be11fd3b1bde7428"
server_key_version: "encryptionkey@0"
I20260928 21:16:23.254585 21561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:23.258940 21561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:23.262080 21561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:23.262218 21561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:23.268738 21561 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42463
I20260928 21:16:23.268746 21628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42463 every 8 connection(s)
I20260928 21:16:23.269667 21561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:23.272264 21629 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:23.273623 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21561
I20260928 21:16:23.273770 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:23.274067 392 external_mini_cluster.cc:1428] Setting key ec6a29377c10a1149f38608a2aa0ff30
I20260928 21:16:23.278578 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c: Bootstrap starting.
I20260928 21:16:23.279148 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:23.275359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:51123 (local address 127.0.0.1:42463)
0928 21:16:23.275759 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:23.275776 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:23.275811 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:16:23.276431 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:23.276447 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:23.276718 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:23.277064 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:23.277854 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:23.277858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:23.277870 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:23.277881 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:23.277934 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:23.277954 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:23.278069 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:23.278173 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:16:23.278351 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":135,"threads_started":1}
I20260928 21:16:23.280704 21629 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:23.281507 21629 log.cc:824] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:23.281829 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:23.283424 21629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c: No bootstrap required, opened a new log
I20260928 21:16:23.285673 21632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:23.283947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:52303 (local address 127.0.0.1:42463)
0928 21:16:23.284072 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:23.284076 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:23.284318 (+ 242us) server_negotiation.cc:407] Connection header received
0928 21:16:23.284441 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:23.284444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:23.284473 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:23.284547 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:23.285267 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:23.285270 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:23.285271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:23.285282 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:23.285304 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:23.285313 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:23.285381 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:23.285488 (+ 107us) server_negotiation.cc:299] Negotiation successful
0928 21:16:23.285528 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:16:23.287554 21629 raft_consensus.cc:348] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795322883b3749c48e03f3c06178b79c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42463 } }
I20260928 21:16:23.287781 21629 raft_consensus.cc:374] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:23.287838 21629 raft_consensus.cc:729] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 795322883b3749c48e03f3c06178b79c, State: Initialized, Role: FOLLOWER
I20260928 21:16:23.288270 21629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795322883b3749c48e03f3c06178b79c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42463 } }
I20260928 21:16:23.288421 21629 raft_consensus.cc:388] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:23.288486 21629 raft_consensus.cc:482] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:23.288580 21629 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:23.289518 21629 raft_consensus.cc:504] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795322883b3749c48e03f3c06178b79c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42463 } }
I20260928 21:16:23.289847 21629 leader_election.cc:302] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 795322883b3749c48e03f3c06178b79c; no voters:
I20260928 21:16:23.290151 21629 leader_election.cc:288] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:23.290262 21639 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:23.290499 21639 raft_consensus.cc:686] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [term 1 LEADER]: Becoming Leader. State: Replica: 795322883b3749c48e03f3c06178b79c, State: Running, Role: LEADER
I20260928 21:16:23.290871 21639 consensus_queue.cc:237] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795322883b3749c48e03f3c06178b79c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42463 } }
I20260928 21:16:23.291406 21629 sys_catalog.cc:564] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:23.292745 21641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 795322883b3749c48e03f3c06178b79c. Latest consensus state: current_term: 1 leader_uuid: "795322883b3749c48e03f3c06178b79c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795322883b3749c48e03f3c06178b79c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42463 } } }
I20260928 21:16:23.292786 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "795322883b3749c48e03f3c06178b79c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "795322883b3749c48e03f3c06178b79c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42463 } } }
I20260928 21:16:23.292899 21641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:23.292918 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:23.293268 21650 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:23.296387 21650 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:23.301911 21650 catalog_manager.cc:1347] Generated new cluster ID: d98f5e9b43e74aac89b5ae8e611684c7
I20260928 21:16:23.301992 21650 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:23.311955 21650 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:23.312268 21650 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:23.331171 21650 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 795322883b3749c48e03f3c06178b79c: Generated new TSK 0
I20260928 21:16:23.331950 21650 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:23.339854 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21561
2026-09-28T21:16:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1388 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:23 dist-test-slave-v7xz krb5kdc[21660](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:23 dist-test-slave-v7xz krb5kdc[21660](info): set up 2 sockets
Sep 28 21:16:23 dist-test-slave-v7xz krb5kdc[21660](info): commencing operation
krb5kdc: starting...
W20260928 21:16:24.389168 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:24 dist-test-slave-v7xz krb5kdc[21660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:24.511490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35453
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35453
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:24.631520 21673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:24.631877 21673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:24.631976 21673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:24.636232 21673 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:24.636368 21673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:24.636408 21673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:24.636466 21673 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:24.636503 21673 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:24.642227 21673 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35453
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:24.643438 21673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:24.644531 21673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:24.653302 21678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:24.653375 21679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:24.653311 21681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:24.653487 21673 server_base.cc:1034] running on GCE node
I20260928 21:16:24.654259 21673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:24.655401 21673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:24.656610 21673 hybrid_clock.cc:648] HybridClock initialized: now 1790630184656596 us; error 29 us; skew 500 ppm
Sep 28 21:16:24 dist-test-slave-v7xz krb5kdc[21660](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:24.659619 21673 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:24.660504 21673 webserver.cc:460] Webserver started at http://127.0.0.1:45383/ using document root <none> and password file <none>
I20260928 21:16:24.661113 21673 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:24.661177 21673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:24.661340 21673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:24.663288 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "1457c82f285c4632893463680810c457"
format_stamp: "Formatted at 2026-09-28 21:16:24 on dist-test-slave-v7xz"
server_key: "ef43e82431f03bfbf1b761c747e72277"
server_key_iv: "abf43e52b1a4a96daac2671d22e660ca"
server_key_version: "encryptionkey@0"
I20260928 21:16:24.663784 21673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "1457c82f285c4632893463680810c457"
format_stamp: "Formatted at 2026-09-28 21:16:24 on dist-test-slave-v7xz"
server_key: "ef43e82431f03bfbf1b761c747e72277"
server_key_iv: "abf43e52b1a4a96daac2671d22e660ca"
server_key_version: "encryptionkey@0"
I20260928 21:16:24.667629 21673 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.004s
I20260928 21:16:24.670050 21688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:24.671362 21673 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:24.671506 21673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "1457c82f285c4632893463680810c457"
format_stamp: "Formatted at 2026-09-28 21:16:24 on dist-test-slave-v7xz"
server_key: "ef43e82431f03bfbf1b761c747e72277"
server_key_iv: "abf43e52b1a4a96daac2671d22e660ca"
server_key_version: "encryptionkey@0"
I20260928 21:16:24.671625 21673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:24.690026 21673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:24.693620 21673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:24.693851 21673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:24.701170 21673 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35453
I20260928 21:16:24.701203 21740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35453 every 8 connection(s)
I20260928 21:16:24.702180 21673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:24.705082 21741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:24.707998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21673
I20260928 21:16:24.708150 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:24.708413 392 external_mini_cluster.cc:1428] Setting key c569c20e1bda11d1db9d4bed6dcd085d
I20260928 21:16:24.711328 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457: Bootstrap starting.
I20260928 21:16:24.714089 21741 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:24.715055 21741 log.cc:824] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:24.717271 21741 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457: No bootstrap required, opened a new log
Sep 28 21:16:24 dist-test-slave-v7xz krb5kdc[21660](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630184, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260928 21:16:24.721608 21741 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457c82f285c4632893463680810c457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260928 21:16:24.721881 21741 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:24.721956 21741 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1457c82f285c4632893463680810c457, State: Initialized, Role: FOLLOWER
I20260928 21:16:24.722476 21741 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457c82f285c4632893463680810c457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260928 21:16:24.722627 21741 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:24.722698 21741 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:24.722798 21741 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:24.723771 21741 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457c82f285c4632893463680810c457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260928 21:16:24.724124 21741 leader_election.cc:302] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1457c82f285c4632893463680810c457; no voters:
I20260928 21:16:24.724438 21741 leader_election.cc:288] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:24.724550 21746 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:24.724748 21746 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [term 1 LEADER]: Becoming Leader. State: Replica: 1457c82f285c4632893463680810c457, State: Running, Role: LEADER
I20260928 21:16:24.724911 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:24.709951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:47099 (local address 127.0.0.1:35453)
0928 21:16:24.710430 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:24.710450 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:24.710486 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:16:24.711113 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:24.711129 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:24.711527 (+ 398us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:24.711823 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:24.712801 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:24.713855 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:24.715096 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:24.715634 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:24.715765 (+ 131us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:24.718580 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:24.718587 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:24.718606 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:24.718640 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:24.721388 (+ 2748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:24.721974 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:24.721979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:24.721984 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:24.722155 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:24.722510 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:24.722513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:24.722515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:24.723065 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:24.723335 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:24.723622 (+ 287us) server_negotiation.cc:299] Negotiation successful
0928 21:16:24.723856 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":169,"threads_started":1}
I20260928 21:16:24.725158 21746 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457c82f285c4632893463680810c457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } }
I20260928 21:16:24.725793 21741 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:24.726399 21748 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1457c82f285c4632893463680810c457. Latest consensus state: current_term: 1 leader_uuid: "1457c82f285c4632893463680810c457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457c82f285c4632893463680810c457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } } }
I20260928 21:16:24.726548 21748 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:24.726823 21747 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1457c82f285c4632893463680810c457" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1457c82f285c4632893463680810c457" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35453 } } }
I20260928 21:16:24.726915 21747 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:24.726982 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:24.728456 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:24.730345 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:24.737213 21750 catalog_manager.cc:1347] Generated new cluster ID: d0232208ff3740fe88e565f6277f6a09
I20260928 21:16:24.737296 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:24.740139 21744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:24.731184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:45111 (local address 127.0.0.1:35453)
0928 21:16:24.731313 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:24.731319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:24.731608 (+ 289us) server_negotiation.cc:407] Connection header received
0928 21:16:24.731719 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:24.731722 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:24.731782 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:24.731873 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:24.732755 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:24.733395 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:24.734239 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:24.734495 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:24.734607 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:24.736327 (+ 1720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:24.736331 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:24.736334 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:24.736364 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:24.738230 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:24.738801 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:24.738805 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:24.738806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:24.738901 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:24.739237 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:24.739240 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:24.739242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:24.739456 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:24.739618 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:24.739873 (+ 255us) server_negotiation.cc:299] Negotiation successful
0928 21:16:24.739976 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:16:24.775876 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:24.777271 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:24.783694 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1457c82f285c4632893463680810c457: Generated new TSK 0
I20260928 21:16:24.784480 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:24.797040 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21673
W20260928 21:16:24.801944 21658 connection.cc:535] client connection to 127.0.0.1:35453 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35453): Connection reset by peer (error 104)
2026-09-28T21:16:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1455 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:24 dist-test-slave-v7xz krb5kdc[21772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:24 dist-test-slave-v7xz krb5kdc[21772](info): set up 2 sockets
Sep 28 21:16:24 dist-test-slave-v7xz krb5kdc[21772](info): commencing operation
krb5kdc: starting...
W20260928 21:16:25.837077 392 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:25 dist-test-slave-v7xz krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:25.964296 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34077
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34077
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:26.075309 21785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:26.075665 21785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:26.075748 21785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:26.079965 21785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:26.080073 21785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:26.080113 21785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:26.080173 21785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:26.080212 21785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:26.085801 21785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34077
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:26.086906 21785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:26.087924 21785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:26.096385 21791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:26.096459 21785 server_base.cc:1034] running on GCE node
W20260928 21:16:26.096370 21790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:26.096370 21793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:26.097237 21785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:26.098315 21785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:26.099502 21785 hybrid_clock.cc:648] HybridClock initialized: now 1790630186099488 us; error 29 us; skew 500 ppm
Sep 28 21:16:26 dist-test-slave-v7xz krb5kdc[21772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:26.102248 21785 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:26.103124 21785 webserver.cc:460] Webserver started at http://127.0.0.1:44657/ using document root <none> and password file <none>
I20260928 21:16:26.103631 21785 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:26.103689 21785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:26.103844 21785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:26.105858 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "0655db6a080e474a85beff044f2ee9b7"
format_stamp: "Formatted at 2026-09-28 21:16:26 on dist-test-slave-v7xz"
server_key: "7553bc722cc696ec894818d9c2620ee9"
server_key_iv: "88d83df7ad2ee17069da0585f4918c92"
server_key_version: "encryptionkey@0"
I20260928 21:16:26.106377 21785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "0655db6a080e474a85beff044f2ee9b7"
format_stamp: "Formatted at 2026-09-28 21:16:26 on dist-test-slave-v7xz"
server_key: "7553bc722cc696ec894818d9c2620ee9"
server_key_iv: "88d83df7ad2ee17069da0585f4918c92"
server_key_version: "encryptionkey@0"
I20260928 21:16:26.109922 21785 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260928 21:16:26.112262 21800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:26.113631 21785 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:26.113780 21785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "0655db6a080e474a85beff044f2ee9b7"
format_stamp: "Formatted at 2026-09-28 21:16:26 on dist-test-slave-v7xz"
server_key: "7553bc722cc696ec894818d9c2620ee9"
server_key_iv: "88d83df7ad2ee17069da0585f4918c92"
server_key_version: "encryptionkey@0"
I20260928 21:16:26.113905 21785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:26.137475 21785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:26.140699 21785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:26.140946 21785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:26.148106 21785 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34077
I20260928 21:16:26.148131 21852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34077 every 8 connection(s)
I20260928 21:16:26.149140 21785 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:26.150771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21785
I20260928 21:16:26.150918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:26.151237 392 external_mini_cluster.cc:1428] Setting key 5f79965806ecbcc6a36232f3e84824c3
I20260928 21:16:26.152500 21853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:26.158949 21853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7: Bootstrap starting.
I20260928 21:16:26.160460 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:26.152653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:48449 (local address 127.0.0.1:34077)
0928 21:16:26.153100 (+ 447us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:26.153117 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:26.153152 (+ 35us) server_negotiation.cc:407] Connection header received
0928 21:16:26.153780 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:26.153796 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:26.154134 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:26.154424 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:26.154761 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:26.155737 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:26.156955 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:26.157464 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:26.157584 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:26.158643 (+ 1059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:26.158648 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:26.158661 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:26.159102 (+ 441us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:26.159180 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:26.159205 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:26.159310 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:26.159568 (+ 258us) server_negotiation.cc:299] Negotiation successful
0928 21:16:26.159731 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":297,"thread_start_us":175,"threads_started":1}
I20260928 21:16:26.161433 21853 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:26.162161 21853 log.cc:824] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:26.163451 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:26.164467 21853 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7: No bootstrap required, opened a new log
I20260928 21:16:26.168740 21853 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0655db6a080e474a85beff044f2ee9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34077 } }
I20260928 21:16:26.169078 21853 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:26.169364 21853 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0655db6a080e474a85beff044f2ee9b7, State: Initialized, Role: FOLLOWER
I20260928 21:16:26.169987 21853 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0655db6a080e474a85beff044f2ee9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34077 } }
I20260928 21:16:26.170212 21853 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:26.170298 21853 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:26.170442 21853 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:26.170610 21856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:26.165694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:53773 (local address 127.0.0.1:34077)
0928 21:16:26.165821 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:26.165826 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:26.165963 (+ 137us) server_negotiation.cc:407] Connection header received
0928 21:16:26.166065 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:26.166067 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:26.166113 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:26.166189 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:26.166426 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:26.167649 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:26.168532 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:26.168901 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:26.169009 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:26.170038 (+ 1029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:26.170041 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:26.170043 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:26.170055 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:26.170080 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:26.170090 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:26.170192 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:26.170380 (+ 188us) server_negotiation.cc:299] Negotiation successful
0928 21:16:26.170421 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:16:26.171591 21853 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0655db6a080e474a85beff044f2ee9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34077 } }
I20260928 21:16:26.171980 21853 leader_election.cc:302] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0655db6a080e474a85beff044f2ee9b7; no voters:
I20260928 21:16:26.172334 21853 leader_election.cc:288] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:26.172410 21863 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:26.172674 21863 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [term 1 LEADER]: Becoming Leader. State: Replica: 0655db6a080e474a85beff044f2ee9b7, State: Running, Role: LEADER
I20260928 21:16:26.173008 21863 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0655db6a080e474a85beff044f2ee9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34077 } }
I20260928 21:16:26.173938 21853 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:26.174898 21865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0655db6a080e474a85beff044f2ee9b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0655db6a080e474a85beff044f2ee9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34077 } } }
I20260928 21:16:26.175029 21865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:26.175266 21864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0655db6a080e474a85beff044f2ee9b7. Latest consensus state: current_term: 1 leader_uuid: "0655db6a080e474a85beff044f2ee9b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0655db6a080e474a85beff044f2ee9b7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34077 } } }
I20260928 21:16:26.175352 21864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:26.175385 21872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:26.178524 21872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:26.184118 21872 catalog_manager.cc:1347] Generated new cluster ID: 0b95705d602548ee869b50fb9bdf1798
I20260928 21:16:26.184190 21872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:26.209232 21872 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:26.210659 21872 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:26.223582 21872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0655db6a080e474a85beff044f2ee9b7: Generated new TSK 0
I20260928 21:16:26.224363 21872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:26.236665 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21785
W20260928 21:16:26.242692 21770 connection.cc:535] client connection to 127.0.0.1:34077 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34077): Connection reset by peer (error 104)
2026-09-28T21:16:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1440 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:26 dist-test-slave-v7xz krb5kdc[21884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:26 dist-test-slave-v7xz krb5kdc[21884](info): set up 2 sockets
Sep 28 21:16:26 dist-test-slave-v7xz krb5kdc[21884](info): commencing operation
krb5kdc: starting...
W20260928 21:16:27.285065 392 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:27 dist-test-slave-v7xz krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630187, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260928 21:16:27.407043 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:27.516880 21897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:27.517155 21897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:27.517208 21897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:27.521129 21897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:27.521188 21897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:27.521207 21897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:27.521224 21897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:27.521245 21897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:27.526496 21897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:27.527657 21897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:27.528656 21897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:16:27.537281 21897 server_base.cc:1034] running on GCE node
W20260928 21:16:27.537227 21905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:27.537251 21902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:27.537220 21903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:27.538110 21897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:27.539230 21897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:27.540390 21897 hybrid_clock.cc:648] HybridClock initialized: now 1790630187540371 us; error 31 us; skew 500 ppm
Sep 28 21:16:27 dist-test-slave-v7xz krb5kdc[21884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630187, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:27.550482 21897 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260928 21:16:27.551407 21897 webserver.cc:460] Webserver started at http://127.0.0.1:42523/ using document root <none> and password file <none>
I20260928 21:16:27.551963 21897 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:27.552026 21897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:27.552232 21897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:27.554414 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "3753881816d845f8bdd5d764453bb10f"
format_stamp: "Formatted at 2026-09-28 21:16:27 on dist-test-slave-v7xz"
server_key: "3986691ae28ba69647b3e658019e724d"
server_key_iv: "b79356a9c78ea61b1bda962cb55d8ffd"
server_key_version: "encryptionkey@0"
I20260928 21:16:27.554932 21897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "3753881816d845f8bdd5d764453bb10f"
format_stamp: "Formatted at 2026-09-28 21:16:27 on dist-test-slave-v7xz"
server_key: "3986691ae28ba69647b3e658019e724d"
server_key_iv: "b79356a9c78ea61b1bda962cb55d8ffd"
server_key_version: "encryptionkey@0"
I20260928 21:16:27.558660 21897 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:16:27.561173 21912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:27.562532 21897 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:27.562655 21897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "3753881816d845f8bdd5d764453bb10f"
format_stamp: "Formatted at 2026-09-28 21:16:27 on dist-test-slave-v7xz"
server_key: "3986691ae28ba69647b3e658019e724d"
server_key_iv: "b79356a9c78ea61b1bda962cb55d8ffd"
server_key_version: "encryptionkey@0"
I20260928 21:16:27.562767 21897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:27.567380 21897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:27.570248 21897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:27.570446 21897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:27.577581 21897 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40825
I20260928 21:16:27.577623 21964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40825 every 8 connection(s)
I20260928 21:16:27.578531 21897 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:27.581277 21965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:27.582823 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 21897
I20260928 21:16:27.582962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:27.583246 392 external_mini_cluster.cc:1428] Setting key 13ac4330c8a18cbc6d99cc722bb45867
I20260928 21:16:27.587910 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f: Bootstrap starting.
I20260928 21:16:27.588820 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:27.584642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:36963 (local address 127.0.0.1:40825)
0928 21:16:27.585117 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:27.585138 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:27.585184 (+ 46us) server_negotiation.cc:407] Connection header received
0928 21:16:27.586031 (+ 847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:27.586054 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:27.586415 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:27.586779 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:27.587487 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:27.587492 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:27.587507 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:27.587521 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:27.587582 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:27.587608 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:27.587741 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:27.587806 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:16:27.587969 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":184,"threads_started":1}
I20260928 21:16:27.590497 21965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:27.591295 21965 log.cc:824] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:27.591550 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:27.593235 21965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f: No bootstrap required, opened a new log
I20260928 21:16:27.595105 21968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:27.593520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.91:40221 (local address 127.0.0.1:40825)
0928 21:16:27.593629 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:27.593635 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:27.593858 (+ 223us) server_negotiation.cc:407] Connection header received
0928 21:16:27.593990 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:27.593992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:27.594019 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:27.594082 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:27.594727 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:27.594730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:27.594732 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:27.594741 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:27.594763 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:27.594779 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:27.594844 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:27.594948 (+ 104us) server_negotiation.cc:299] Negotiation successful
0928 21:16:27.594977 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260928 21:16:27.597555 21965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3753881816d845f8bdd5d764453bb10f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260928 21:16:27.597817 21965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:27.597872 21965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3753881816d845f8bdd5d764453bb10f, State: Initialized, Role: FOLLOWER
I20260928 21:16:27.598255 21965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3753881816d845f8bdd5d764453bb10f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260928 21:16:27.598381 21965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:27.598420 21965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:27.598482 21965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:27.599288 21965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3753881816d845f8bdd5d764453bb10f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260928 21:16:27.599576 21965 leader_election.cc:302] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3753881816d845f8bdd5d764453bb10f; no voters:
I20260928 21:16:27.599810 21965 leader_election.cc:288] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:27.599946 21975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:27.600180 21975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [term 1 LEADER]: Becoming Leader. State: Replica: 3753881816d845f8bdd5d764453bb10f, State: Running, Role: LEADER
I20260928 21:16:27.600544 21975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3753881816d845f8bdd5d764453bb10f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } }
I20260928 21:16:27.601136 21965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:27.602329 21977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3753881816d845f8bdd5d764453bb10f. Latest consensus state: current_term: 1 leader_uuid: "3753881816d845f8bdd5d764453bb10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3753881816d845f8bdd5d764453bb10f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } } }
I20260928 21:16:27.602348 21976 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3753881816d845f8bdd5d764453bb10f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3753881816d845f8bdd5d764453bb10f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40825 } } }
I20260928 21:16:27.602499 21977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:27.602514 21976 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:27.602954 21984 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:27.605703 21984 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:27.611209 21984 catalog_manager.cc:1347] Generated new cluster ID: f38fbaa85c624202b50d2e9183d05d63
I20260928 21:16:27.611286 21984 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:27.634349 21984 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:27.634693 21984 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:27.645711 21984 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3753881816d845f8bdd5d764453bb10f: Generated new TSK 0
I20260928 21:16:27.646436 21984 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:27.655614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 21897
2026-09-28T21:16:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:27 dist-test-slave-v7xz krb5kdc[21996](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:27 dist-test-slave-v7xz krb5kdc[21996](info): set up 2 sockets
Sep 28 21:16:27 dist-test-slave-v7xz krb5kdc[21996](info): commencing operation
krb5kdc: starting...
W20260928 21:16:28.689159 392 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:28 dist-test-slave-v7xz krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:28.808266 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:28.917574 22009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:28.917850 22009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:28.917892 22009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:28.922101 22009 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:28.922192 22009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:28.922212 22009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:28.922230 22009 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:28.922246 22009 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:28.927806 22009 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44959
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:28.928968 22009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:28.929973 22009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:28.938956 22014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:28.938920 22015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:28.938920 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:28.939314 22009 server_base.cc:1034] running on GCE node
I20260928 21:16:28.939898 22009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:28.941079 22009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:28.942281 22009 hybrid_clock.cc:648] HybridClock initialized: now 1790630188942246 us; error 50 us; skew 500 ppm
Sep 28 21:16:28 dist-test-slave-v7xz krb5kdc[21996](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:28.945153 22009 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:28.946115 22009 webserver.cc:460] Webserver started at http://127.0.98.62:38323/ using document root <none> and password file <none>
I20260928 21:16:28.946702 22009 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:28.946795 22009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:28.947033 22009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:28.949219 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "0b071914d00e406a841e5654a7da1fe1"
format_stamp: "Formatted at 2026-09-28 21:16:28 on dist-test-slave-v7xz"
server_key: "b2d9430caa1cb8897ec3859d4464e1c0"
server_key_iv: "95059a7621d6cbaaa56bea51292e3f7f"
server_key_version: "encryptionkey@0"
I20260928 21:16:28.949779 22009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b071914d00e406a841e5654a7da1fe1"
format_stamp: "Formatted at 2026-09-28 21:16:28 on dist-test-slave-v7xz"
server_key: "b2d9430caa1cb8897ec3859d4464e1c0"
server_key_iv: "95059a7621d6cbaaa56bea51292e3f7f"
server_key_version: "encryptionkey@0"
I20260928 21:16:28.953495 22009 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:28.955935 22024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:28.957297 22009 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:28.957430 22009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "0b071914d00e406a841e5654a7da1fe1"
format_stamp: "Formatted at 2026-09-28 21:16:28 on dist-test-slave-v7xz"
server_key: "b2d9430caa1cb8897ec3859d4464e1c0"
server_key_iv: "95059a7621d6cbaaa56bea51292e3f7f"
server_key_version: "encryptionkey@0"
I20260928 21:16:28.957542 22009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:28.971671 22009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:28.975278 22009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:28.975487 22009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:28.982545 22009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44959
I20260928 21:16:28.982570 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44959 every 8 connection(s)
I20260928 21:16:28.983505 22009 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:28.984383 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22009
I20260928 21:16:28.984545 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:28.984905 392 external_mini_cluster.cc:1428] Setting key 98f36926803692a354e9afb76e4ecbea
I20260928 21:16:28.987071 22077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:28.993037 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1: Bootstrap starting.
Sep 28 21:16:28 dist-test-slave-v7xz krb5kdc[21996](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:28.995715 22077 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:28.996526 22077 log.cc:824] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:28.998827 22077 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1: No bootstrap required, opened a new log
I20260928 21:16:28.999493 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:28.986406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43207 (local address 127.0.98.62:44959)
0928 21:16:28.986864 (+ 458us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:28.986879 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:28.986915 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:16:28.987597 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:28.987623 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:28.987972 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:28.988308 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:28.989188 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:28.990167 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:28.991031 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:28.991492 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:28.991671 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:28.994595 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:28.994600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:28.994618 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:28.994645 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:28.996379 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:28.997005 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:28.997012 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:28.997018 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:28.997165 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:28.997644 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:28.997647 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:28.997648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:28.998007 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:28.998170 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:28.998493 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 21:16:28.998736 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":167,"threads_started":1}
I20260928 21:16:29.002570 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:29.003127 22077 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b071914d00e406a841e5654a7da1fe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260928 21:16:29.003422 22077 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:29.003489 22077 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b071914d00e406a841e5654a7da1fe1, State: Initialized, Role: FOLLOWER
I20260928 21:16:29.003988 22077 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b071914d00e406a841e5654a7da1fe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260928 21:16:29.004160 22077 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:29.004221 22077 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:29.004320 22077 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:29.005595 22077 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b071914d00e406a841e5654a7da1fe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260928 21:16:29.005963 22077 leader_election.cc:302] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b071914d00e406a841e5654a7da1fe1; no voters:
I20260928 21:16:29.006305 22077 leader_election.cc:288] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:29.006417 22087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:29.006647 22087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [term 1 LEADER]: Becoming Leader. State: Replica: 0b071914d00e406a841e5654a7da1fe1, State: Running, Role: LEADER
I20260928 21:16:29.007010 22087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b071914d00e406a841e5654a7da1fe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } }
I20260928 21:16:29.007609 22077 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:29.008734 22089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b071914d00e406a841e5654a7da1fe1. Latest consensus state: current_term: 1 leader_uuid: "0b071914d00e406a841e5654a7da1fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b071914d00e406a841e5654a7da1fe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260928 21:16:29.008893 22089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:29.009245 22088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b071914d00e406a841e5654a7da1fe1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b071914d00e406a841e5654a7da1fe1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44959 } } }
I20260928 21:16:29.009351 22088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1 [sys.catalog]: This master's current role is: LEADER
W20260928 21:16:29.013367 22102 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:29.013504 22102 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:29.013985 22080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:29.004823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55263 (local address 127.0.98.62:44959)
0928 21:16:29.004967 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:29.004971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:29.005144 (+ 173us) server_negotiation.cc:407] Connection header received
0928 21:16:29.005237 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:29.005243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:29.005291 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:29.005385 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:29.006086 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:29.006912 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:29.008092 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:29.008479 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:29.008613 (+ 134us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:29.010597 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:29.010600 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:29.010603 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:29.010624 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:29.012104 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:29.012591 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:29.012595 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:29.012598 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:29.012674 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:29.013103 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:29.013106 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:29.013108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:29.013289 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:29.013444 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:29.013729 (+ 285us) server_negotiation.cc:299] Negotiation successful
0928 21:16:29.013831 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260928 21:16:29.014143 22096 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:29.015097 22096 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:29.021095 22096 catalog_manager.cc:1347] Generated new cluster ID: f46f7277a0ac499ba61b15edb8ce9702
I20260928 21:16:29.021184 22096 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:29.035856 22096 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:29.037256 22096 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:29.046664 22096 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b071914d00e406a841e5654a7da1fe1: Generated new TSK 0
I20260928 21:16:29.047435 22096 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:29.059748 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22009
W20260928 21:16:29.065222 21994 connection.cc:535] client connection to 127.0.98.62:44959 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44959): Connection reset by peer (error 104)
2026-09-28T21:16:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:29 dist-test-slave-v7xz krb5kdc[22108](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:29 dist-test-slave-v7xz krb5kdc[22108](info): set up 2 sockets
Sep 28 21:16:29 dist-test-slave-v7xz krb5kdc[22108](info): commencing operation
krb5kdc: starting...
W20260928 21:16:30.081218 392 mini_kdc.cc:121] Time spent starting KDC: real 1.002s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:30 dist-test-slave-v7xz krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:30.199958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42023
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:30.310693 22121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:30.310966 22121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:30.311007 22121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:30.314966 22121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:30.315032 22121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:30.315052 22121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:30.315068 22121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:30.315083 22121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:30.320958 22121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42023
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:30.321985 22121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:30.323001 22121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:30.331702 22127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:30.331722 22129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:30.331756 22126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:30.332099 22121 server_base.cc:1034] running on GCE node
I20260928 21:16:30.332665 22121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:30.333881 22121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:30.335078 22121 hybrid_clock.cc:648] HybridClock initialized: now 1790630190335060 us; error 61 us; skew 500 ppm
Sep 28 21:16:30 dist-test-slave-v7xz krb5kdc[22108](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630190, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:30.337749 22121 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:30.338603 22121 webserver.cc:460] Webserver started at http://127.0.98.62:34413/ using document root <none> and password file <none>
I20260928 21:16:30.339115 22121 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:30.339171 22121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:30.339331 22121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:30.341389 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "b33b9e33d7c74c278729bfe8f68a6af0"
format_stamp: "Formatted at 2026-09-28 21:16:30 on dist-test-slave-v7xz"
server_key: "17e3c2dadf29d046bc11ef32d839fbd2"
server_key_iv: "1860076af01329b31d2f897a7e02f787"
server_key_version: "encryptionkey@0"
I20260928 21:16:30.341885 22121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "b33b9e33d7c74c278729bfe8f68a6af0"
format_stamp: "Formatted at 2026-09-28 21:16:30 on dist-test-slave-v7xz"
server_key: "17e3c2dadf29d046bc11ef32d839fbd2"
server_key_iv: "1860076af01329b31d2f897a7e02f787"
server_key_version: "encryptionkey@0"
I20260928 21:16:30.345404 22121 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:30.347757 22136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:30.348950 22121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:30.349074 22121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "b33b9e33d7c74c278729bfe8f68a6af0"
format_stamp: "Formatted at 2026-09-28 21:16:30 on dist-test-slave-v7xz"
server_key: "17e3c2dadf29d046bc11ef32d839fbd2"
server_key_iv: "1860076af01329b31d2f897a7e02f787"
server_key_version: "encryptionkey@0"
I20260928 21:16:30.349174 22121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:30.380861 22121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:30.383731 22121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:30.383944 22121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:30.391151 22121 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42023
I20260928 21:16:30.391176 22188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42023 every 8 connection(s)
I20260928 21:16:30.392095 22121 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:30.394888 22189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:30.396637 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22121
I20260928 21:16:30.396858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:30.397171 392 external_mini_cluster.cc:1428] Setting key 3dc9e8f0f503fa6c963bc518f213d1f8
I20260928 21:16:30.401398 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0: Bootstrap starting.
I20260928 21:16:30.403663 22189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:30.404539 22189 log.cc:824] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:30.406420 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:30.398627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52495 (local address 127.0.98.62:42023)
0928 21:16:30.399089 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:30.399104 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:30.399140 (+ 36us) server_negotiation.cc:407] Connection header received
0928 21:16:30.399769 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:30.399785 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:30.400129 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:30.400474 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:30.400957 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:30.401983 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:30.402894 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:30.403377 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:30.403554 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:30.404475 (+ 921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:30.404480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:30.404493 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:30.405023 (+ 530us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:30.405094 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:30.405128 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:30.405274 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:30.405491 (+ 217us) server_negotiation.cc:299] Negotiation successful
0928 21:16:30.405648 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":139,"threads_started":1}
I20260928 21:16:30.406783 22189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0: No bootstrap required, opened a new log
I20260928 21:16:30.409484 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:30.411455 22189 raft_consensus.cc:348] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } }
I20260928 21:16:30.411743 22189 raft_consensus.cc:374] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:30.411852 22189 raft_consensus.cc:729] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b33b9e33d7c74c278729bfe8f68a6af0, State: Initialized, Role: FOLLOWER
I20260928 21:16:30.412353 22189 consensus_queue.cc:260] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } }
I20260928 21:16:30.412509 22189 raft_consensus.cc:388] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:30.412590 22189 raft_consensus.cc:482] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:30.412691 22189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:30.413672 22189 raft_consensus.cc:504] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } }
I20260928 21:16:30.414018 22189 leader_election.cc:302] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b33b9e33d7c74c278729bfe8f68a6af0; no voters:
I20260928 21:16:30.414364 22189 leader_election.cc:288] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:30.414433 22199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:30.414639 22199 raft_consensus.cc:686] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [term 1 LEADER]: Becoming Leader. State: Replica: b33b9e33d7c74c278729bfe8f68a6af0, State: Running, Role: LEADER
I20260928 21:16:30.414996 22199 consensus_queue.cc:237] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } }
I20260928 21:16:30.415755 22189 sys_catalog.cc:564] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:30.416164 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:30.411797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50605 (local address 127.0.98.62:42023)
0928 21:16:30.411952 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:30.411959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:30.412141 (+ 182us) server_negotiation.cc:407] Connection header received
0928 21:16:30.412290 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:30.412293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:30.412339 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:30.412423 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:30.412717 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:30.413311 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:30.414242 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:30.414579 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:30.414693 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:30.415666 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:30.415669 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:30.415672 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:30.415684 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:30.415715 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:30.415725 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:30.415821 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:30.415957 (+ 136us) server_negotiation.cc:299] Negotiation successful
0928 21:16:30.415992 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260928 21:16:30.416555 22199 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b33b9e33d7c74c278729bfe8f68a6af0. Latest consensus state: current_term: 1 leader_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } }
I20260928 21:16:30.416689 22199 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:30.417115 22208 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:30.417702 22200 sys_catalog.cc:455] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b33b9e33d7c74c278729bfe8f68a6af0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42023 } } }
I20260928 21:16:30.417798 22200 sys_catalog.cc:458] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:30.420480 22208 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:30.427217 22208 catalog_manager.cc:1347] Generated new cluster ID: 9ac58200c6144dbba1891ba7cb1da320
I20260928 21:16:30.427306 22208 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:30.438570 22208 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:30.439463 22208 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:30.444871 22208 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b33b9e33d7c74c278729bfe8f68a6af0: Generated new TSK 0
I20260928 21:16:30.445662 22208 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:30.459987 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22121
W20260928 21:16:30.464708 22106 connection.cc:535] client connection to 127.0.98.62:42023 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:42023): Connection reset by peer (error 104)
2026-09-28T21:16:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:30 dist-test-slave-v7xz krb5kdc[22220](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:30 dist-test-slave-v7xz krb5kdc[22220](info): set up 2 sockets
Sep 28 21:16:30 dist-test-slave-v7xz krb5kdc[22220](info): commencing operation
krb5kdc: starting...
W20260928 21:16:31.513075 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:31 dist-test-slave-v7xz krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630191, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:31.635110 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36421
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39059
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36421
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:31.744796 22233 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:31.745149 22233 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:31.745229 22233 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:31.749326 22233 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:31.749423 22233 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:31.749462 22233 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:31.749522 22233 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:31.749559 22233 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:31.754952 22233 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39059
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36421
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22233
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:31.756112 22233 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:31.757222 22233 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:31.766010 22241 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:31.766018 22239 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:31.766018 22238 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:31.766338 22233 server_base.cc:1034] running on GCE node
I20260928 21:16:31.766920 22233 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:31.768016 22233 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:31.769204 22233 hybrid_clock.cc:648] HybridClock initialized: now 1790630191769186 us; error 30 us; skew 500 ppm
Sep 28 21:16:31 dist-test-slave-v7xz krb5kdc[22220](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:31.772035 22233 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:31.773387 22233 webserver.cc:460] Webserver started at http://127.0.98.62:45253/ using document root <none> and password file <none>
I20260928 21:16:31.773957 22233 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:31.774057 22233 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:31.774287 22233 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:31.776291 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "6ce451dcab2d48e697cdaecb31d1ef3a"
format_stamp: "Formatted at 2026-09-28 21:16:31 on dist-test-slave-v7xz"
server_key: "e7557af929764a08529601045ab01d50"
server_key_iv: "4278b6cf8075e9142b95f40c4e6c2aac"
server_key_version: "encryptionkey@0"
I20260928 21:16:31.776894 22233 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "6ce451dcab2d48e697cdaecb31d1ef3a"
format_stamp: "Formatted at 2026-09-28 21:16:31 on dist-test-slave-v7xz"
server_key: "e7557af929764a08529601045ab01d50"
server_key_iv: "4278b6cf8075e9142b95f40c4e6c2aac"
server_key_version: "encryptionkey@0"
I20260928 21:16:31.780452 22233 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:31.783139 22248 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:31.784338 22233 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:31.784482 22233 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "6ce451dcab2d48e697cdaecb31d1ef3a"
format_stamp: "Formatted at 2026-09-28 21:16:31 on dist-test-slave-v7xz"
server_key: "e7557af929764a08529601045ab01d50"
server_key_iv: "4278b6cf8075e9142b95f40c4e6c2aac"
server_key_version: "encryptionkey@0"
I20260928 21:16:31.784600 22233 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:31.789186 22233 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:31.792351 22233 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:31.792573 22233 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:31.799393 22233 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36421
I20260928 21:16:31.799417 22300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36421 every 8 connection(s)
I20260928 21:16:31.800254 22233 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:31.800953 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22233
I20260928 21:16:31.801111 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:31.801410 392 external_mini_cluster.cc:1428] Setting key cd7f50d3035c602278bc2b2e709a377a
I20260928 21:16:31.803530 22301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:31.806708 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:31.802828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54599 (local address 127.0.98.62:36421)
0928 21:16:31.803229 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:31.803242 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:31.803466 (+ 224us) server_negotiation.cc:407] Connection header received
0928 21:16:31.804195 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:31.804211 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:31.804472 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:31.804787 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:31.805547 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:31.805552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:31.805564 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:31.805572 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:31.805614 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:31.805631 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:31.805735 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:31.805847 (+ 112us) server_negotiation.cc:299] Negotiation successful
0928 21:16:31.806005 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":118,"threads_started":1}
I20260928 21:16:31.809643 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:31.809973 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a: Bootstrap starting.
I20260928 21:16:31.812669 22301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:31.813668 22301 log.cc:824] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:31.813673 22304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:31.811987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52155 (local address 127.0.98.62:36421)
0928 21:16:31.812110 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:31.812115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:31.812388 (+ 273us) server_negotiation.cc:407] Connection header received
0928 21:16:31.812426 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:31.812427 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:31.812455 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:31.812516 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:31.813248 (+ 732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:31.813250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:31.813252 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:31.813262 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:31.813284 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:31.813294 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:31.813362 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:31.813470 (+ 108us) server_negotiation.cc:299] Negotiation successful
0928 21:16:31.813501 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:16:31.815866 22301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a: No bootstrap required, opened a new log
I20260928 21:16:31.820015 22301 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } }
I20260928 21:16:31.820248 22301 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:31.820406 22301 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce451dcab2d48e697cdaecb31d1ef3a, State: Initialized, Role: FOLLOWER
I20260928 21:16:31.821022 22301 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } }
I20260928 21:16:31.821157 22301 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:31.821244 22301 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:31.821341 22301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:31.822304 22301 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } }
I20260928 21:16:31.822652 22301 leader_election.cc:302] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6ce451dcab2d48e697cdaecb31d1ef3a; no voters:
I20260928 21:16:31.822966 22301 leader_election.cc:288] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:31.823091 22311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:31.823359 22311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [term 1 LEADER]: Becoming Leader. State: Replica: 6ce451dcab2d48e697cdaecb31d1ef3a, State: Running, Role: LEADER
I20260928 21:16:31.823747 22311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } }
I20260928 21:16:31.824270 22301 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:31.825548 22313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6ce451dcab2d48e697cdaecb31d1ef3a. Latest consensus state: current_term: 1 leader_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } }
I20260928 21:16:31.825501 22312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6ce451dcab2d48e697cdaecb31d1ef3a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36421 } } }
I20260928 21:16:31.825639 22313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:31.825639 22312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a [sys.catalog]: This master's current role is: LEADER
W20260928 21:16:31.828935 22326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:31.829016 22326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:31.829097 22321 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:31.829777 22321 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:31.835702 22321 catalog_manager.cc:1347] Generated new cluster ID: 79e77dc5fc164f9da54a76fd3e68ea1f
I20260928 21:16:31.835795 22321 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:31.855578 22321 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:31.855942 22321 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:31.871984 22321 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6ce451dcab2d48e697cdaecb31d1ef3a: Generated new TSK 0
I20260928 21:16:31.872782 22321 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:31.882441 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22233
2026-09-28T21:16:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1422 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:31 dist-test-slave-v7xz krb5kdc[22332](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:31 dist-test-slave-v7xz krb5kdc[22332](info): set up 2 sockets
Sep 28 21:16:31 dist-test-slave-v7xz krb5kdc[22332](info): commencing operation
krb5kdc: starting...
W20260928 21:16:32.917035 392 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:32 dist-test-slave-v7xz krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:33.036262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42117
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:33.144335 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:33.144667 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:33.144752 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:33.148900 22345 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:33.149003 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:33.149051 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:33.149087 22345 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:33.149119 22345 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:33.154418 22345 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42117
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:33.155509 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:33.156546 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:33.164860 22353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:33.164896 22351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:33.164863 22350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:33.165143 22345 server_base.cc:1034] running on GCE node
I20260928 21:16:33.165753 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:33.166775 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:33.167935 22345 hybrid_clock.cc:648] HybridClock initialized: now 1790630193167906 us; error 43 us; skew 500 ppm
Sep 28 21:16:33 dist-test-slave-v7xz krb5kdc[22332](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:33.175441 22345 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:33.176481 22345 webserver.cc:460] Webserver started at http://127.0.98.62:41133/ using document root <none> and password file <none>
I20260928 21:16:33.177156 22345 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:33.177258 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:33.177495 22345 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:33.179631 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "3654a9e767474f9e825d939f3a8dca59"
format_stamp: "Formatted at 2026-09-28 21:16:33 on dist-test-slave-v7xz"
server_key: "a299033e989d7474eed8b57698f69a26"
server_key_iv: "b0b0dcfb659e440fbee488e3489594c0"
server_key_version: "encryptionkey@0"
I20260928 21:16:33.180192 22345 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "3654a9e767474f9e825d939f3a8dca59"
format_stamp: "Formatted at 2026-09-28 21:16:33 on dist-test-slave-v7xz"
server_key: "a299033e989d7474eed8b57698f69a26"
server_key_iv: "b0b0dcfb659e440fbee488e3489594c0"
server_key_version: "encryptionkey@0"
I20260928 21:16:33.183864 22345 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260928 21:16:33.186507 22360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:33.187747 22345 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:33.187901 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "3654a9e767474f9e825d939f3a8dca59"
format_stamp: "Formatted at 2026-09-28 21:16:33 on dist-test-slave-v7xz"
server_key: "a299033e989d7474eed8b57698f69a26"
server_key_iv: "b0b0dcfb659e440fbee488e3489594c0"
server_key_version: "encryptionkey@0"
I20260928 21:16:33.188023 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:33.215405 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:33.218745 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:33.218957 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:33.225965 22345 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42117
I20260928 21:16:33.225991 22412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42117 every 8 connection(s)
I20260928 21:16:33.226863 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:33.229573 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:33.232674 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22345
I20260928 21:16:33.232828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:33.233124 392 external_mini_cluster.cc:1428] Setting key 88b32914b2b75e5ec4f29f5cb2dcb00c
I20260928 21:16:33.235327 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59: Bootstrap starting.
I20260928 21:16:33.237551 22413 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:33.238286 22413 log.cc:824] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:33.240124 22413 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59: No bootstrap required, opened a new log
Sep 28 21:16:33 dist-test-slave-v7xz krb5kdc[22332](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630192, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:33.244410 22413 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3654a9e767474f9e825d939f3a8dca59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42117 } }
I20260928 21:16:33.244627 22413 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:33.244704 22413 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3654a9e767474f9e825d939f3a8dca59, State: Initialized, Role: FOLLOWER
I20260928 21:16:33.245211 22413 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3654a9e767474f9e825d939f3a8dca59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42117 } }
I20260928 21:16:33.245375 22413 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:33.245471 22413 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:33.245602 22413 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:33.246476 22413 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3654a9e767474f9e825d939f3a8dca59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42117 } }
I20260928 21:16:33.246834 22413 leader_election.cc:302] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3654a9e767474f9e825d939f3a8dca59; no voters:
I20260928 21:16:33.247179 22413 leader_election.cc:288] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:33.247246 22418 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:33.247460 22418 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [term 1 LEADER]: Becoming Leader. State: Replica: 3654a9e767474f9e825d939f3a8dca59, State: Running, Role: LEADER
I20260928 21:16:33.247721 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:33.234533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33571 (local address 127.0.98.62:42117)
0928 21:16:33.234962 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:33.234978 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:33.235011 (+ 33us) server_negotiation.cc:407] Connection header received
0928 21:16:33.235680 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:33.235699 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:33.236023 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:33.236357 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:33.237217 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:33.238255 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:33.239337 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:33.239800 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:33.239979 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:33.242887 (+ 2908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:33.242892 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:33.242905 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:33.242930 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:33.244591 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:33.245106 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:33.245113 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:33.245123 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:33.245232 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:33.245551 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:33.245558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:33.245561 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:33.245995 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:33.246217 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:33.246451 (+ 234us) server_negotiation.cc:299] Negotiation successful
0928 21:16:33.246771 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":132,"threads_started":1}
I20260928 21:16:33.247838 22418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3654a9e767474f9e825d939f3a8dca59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42117 } }
I20260928 21:16:33.248490 22413 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:33.250419 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3654a9e767474f9e825d939f3a8dca59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3654a9e767474f9e825d939f3a8dca59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42117 } } }
I20260928 21:16:33.250773 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:33.250676 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3654a9e767474f9e825d939f3a8dca59. Latest consensus state: current_term: 1 leader_uuid: "3654a9e767474f9e825d939f3a8dca59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3654a9e767474f9e825d939f3a8dca59" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42117 } } }
I20260928 21:16:33.251034 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:33.251375 22432 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:33.251125 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:33.254218 22432 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:33.259735 22432 catalog_manager.cc:1347] Generated new cluster ID: fc73510663c44cffb45781fd6aa73f69
I20260928 21:16:33.259835 22432 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:33.262097 22416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:33.253440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41733 (local address 127.0.98.62:42117)
0928 21:16:33.253550 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:33.253556 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:33.253570 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:33.253604 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:33.253606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:33.253649 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:33.253737 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:33.254495 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:33.255255 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:33.256394 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:33.256721 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:33.256857 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:33.258467 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:33.258471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:33.258474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:33.258498 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:33.260235 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:33.260723 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:33.260727 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:33.260729 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:33.260809 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:33.261155 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:33.261158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:33.261160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:33.261359 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:33.261457 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:33.261827 (+ 370us) server_negotiation.cc:299] Negotiation successful
0928 21:16:33.261944 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260928 21:16:33.275894 22432 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:33.277145 22432 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:33.285221 22432 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3654a9e767474f9e825d939f3a8dca59: Generated new TSK 0
I20260928 21:16:33.286055 22432 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:33.289881 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22345
2026-09-28T21:16:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:33 dist-test-slave-v7xz krb5kdc[22444](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:33 dist-test-slave-v7xz krb5kdc[22444](info): set up 2 sockets
Sep 28 21:16:33 dist-test-slave-v7xz krb5kdc[22444](info): commencing operation
krb5kdc: starting...
W20260928 21:16:34.325075 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:34 dist-test-slave-v7xz krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630194, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:34.452821 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33957
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33957
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:34.563468 22457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:34.563808 22457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:34.563889 22457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:34.567950 22457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:34.568055 22457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:34.568095 22457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:34.568132 22457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:34.568166 22457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:34.574170 22457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33957
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:34.575634 22457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:34.576924 22457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:34.588042 22463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:34.588042 22462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:34.588277 22465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:34.588449 22457 server_base.cc:1034] running on GCE node
I20260928 21:16:34.589164 22457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:34.590343 22457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:34.591516 22457 hybrid_clock.cc:648] HybridClock initialized: now 1790630194591499 us; error 31 us; skew 500 ppm
Sep 28 21:16:34 dist-test-slave-v7xz krb5kdc[22444](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630194, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:34.594265 22457 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:34.595206 22457 webserver.cc:460] Webserver started at http://127.0.98.62:36119/ using document root <none> and password file <none>
I20260928 21:16:34.595767 22457 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:34.595825 22457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:34.596057 22457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:34.598134 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "2c8a3d7a25c44c1886f0f6db11c850ea"
format_stamp: "Formatted at 2026-09-28 21:16:34 on dist-test-slave-v7xz"
server_key: "9d4f31013ccc0691f67937e6e86d3ab7"
server_key_iv: "1bedbcc12f20b4fa504c9b342afb4557"
server_key_version: "encryptionkey@0"
I20260928 21:16:34.598699 22457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "2c8a3d7a25c44c1886f0f6db11c850ea"
format_stamp: "Formatted at 2026-09-28 21:16:34 on dist-test-slave-v7xz"
server_key: "9d4f31013ccc0691f67937e6e86d3ab7"
server_key_iv: "1bedbcc12f20b4fa504c9b342afb4557"
server_key_version: "encryptionkey@0"
I20260928 21:16:34.602244 22457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:34.604560 22472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:34.605787 22457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:34.605926 22457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "2c8a3d7a25c44c1886f0f6db11c850ea"
format_stamp: "Formatted at 2026-09-28 21:16:34 on dist-test-slave-v7xz"
server_key: "9d4f31013ccc0691f67937e6e86d3ab7"
server_key_iv: "1bedbcc12f20b4fa504c9b342afb4557"
server_key_version: "encryptionkey@0"
I20260928 21:16:34.606036 22457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:34.624053 22457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:34.627730 22457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:34.627956 22457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:34.637431 22457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33957
I20260928 21:16:34.637446 22524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33957 every 8 connection(s)
I20260928 21:16:34.638584 22457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:34.639312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22457
I20260928 21:16:34.639472 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:34.639762 392 external_mini_cluster.cc:1428] Setting key b7651b2b16e62cbbdc531dccc247109d
I20260928 21:16:34.641709 22525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:34.648017 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea: Bootstrap starting.
I20260928 21:16:34.649102 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:34.641161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39117 (local address 127.0.98.62:33957)
0928 21:16:34.641565 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:34.641586 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:34.641956 (+ 370us) server_negotiation.cc:407] Connection header received
0928 21:16:34.642607 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:34.642623 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:34.642977 (+ 354us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:34.643322 (+ 345us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:34.643730 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:34.644700 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:34.645755 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:34.646236 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:34.646413 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:34.647340 (+ 927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:34.647346 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:34.647363 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:34.647747 (+ 384us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:34.647801 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:34.647824 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:34.647946 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:34.648067 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:16:34.648286 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":254,"thread_start_us":131,"threads_started":1}
I20260928 21:16:34.650482 22525 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:34.651319 22525 log.cc:824] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:34.652061 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:34.653699 22525 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea: No bootstrap required, opened a new log
I20260928 21:16:34.658373 22525 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33957 } }
I20260928 21:16:34.658617 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:34.654412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42763 (local address 127.0.98.62:33957)
0928 21:16:34.654535 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:34.654540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:34.654776 (+ 236us) server_negotiation.cc:407] Connection header received
0928 21:16:34.654876 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:34.654879 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:34.654923 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:34.654998 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:34.655231 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:34.656038 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:34.656909 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:34.657259 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:34.657377 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:34.658185 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:34.658188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:34.658191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:34.658204 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:34.658231 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:34.658243 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:34.658321 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:34.658437 (+ 116us) server_negotiation.cc:299] Negotiation successful
0928 21:16:34.658494 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260928 21:16:34.658671 22525 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:34.658722 22525 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c8a3d7a25c44c1886f0f6db11c850ea, State: Initialized, Role: FOLLOWER
I20260928 21:16:34.659183 22525 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33957 } }
I20260928 21:16:34.659312 22525 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:34.659401 22525 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:34.659503 22525 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:34.660416 22525 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33957 } }
I20260928 21:16:34.660768 22525 leader_election.cc:302] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c8a3d7a25c44c1886f0f6db11c850ea; no voters:
I20260928 21:16:34.661130 22525 leader_election.cc:288] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:34.661283 22535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:34.661536 22535 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [term 1 LEADER]: Becoming Leader. State: Replica: 2c8a3d7a25c44c1886f0f6db11c850ea, State: Running, Role: LEADER
I20260928 21:16:34.661895 22535 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33957 } }
I20260928 21:16:34.662482 22525 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:34.663744 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33957 } } }
I20260928 21:16:34.663875 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:34.664049 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c8a3d7a25c44c1886f0f6db11c850ea. Latest consensus state: current_term: 1 leader_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c8a3d7a25c44c1886f0f6db11c850ea" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33957 } } }
I20260928 21:16:34.664290 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea [sys.catalog]: This master's current role is: LEADER
W20260928 21:16:34.667508 22550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:34.667572 22550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:34.667658 22544 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:34.668334 22544 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:34.673336 22544 catalog_manager.cc:1347] Generated new cluster ID: fa2655aba9c2443697e8bde445836319
I20260928 21:16:34.673404 22544 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:34.688373 22544 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:34.689765 22544 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:34.695927 22544 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c8a3d7a25c44c1886f0f6db11c850ea: Generated new TSK 0
I20260928 21:16:34.696743 22544 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:34.701241 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22457
2026-09-28T21:16:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1411 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:34 dist-test-slave-v7xz krb5kdc[22556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:34 dist-test-slave-v7xz krb5kdc[22556](info): set up 2 sockets
Sep 28 21:16:34 dist-test-slave-v7xz krb5kdc[22556](info): commencing operation
krb5kdc: starting...
W20260928 21:16:35.741106 392 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:35 dist-test-slave-v7xz krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:35.864015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38793
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:35.974908 22569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:35.975175 22569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:35.975217 22569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:35.979097 22569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:35.979152 22569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:35.979177 22569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:35.979194 22569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:35.979213 22569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:35.984534 22569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38793
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:35.985622 22569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:35.986654 22569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:35.995285 22575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:35.995491 22569 server_base.cc:1034] running on GCE node
W20260928 21:16:35.995301 22574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:35.995286 22577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:35.996209 22569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:35.997412 22569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:35.998633 22569 hybrid_clock.cc:648] HybridClock initialized: now 1790630195998617 us; error 33 us; skew 500 ppm
Sep 28 21:16:35 dist-test-slave-v7xz krb5kdc[22556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630195, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:36.001410 22569 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:36.002316 22569 webserver.cc:460] Webserver started at http://127.0.98.62:40125/ using document root <none> and password file <none>
I20260928 21:16:36.002831 22569 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:36.002888 22569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:36.003043 22569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:36.005062 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "3d4badc0d65b47c69ab5d1e95df1f643"
format_stamp: "Formatted at 2026-09-28 21:16:35 on dist-test-slave-v7xz"
server_key: "6fdcbd990c5def9c703d0fff5140aea6"
server_key_iv: "724dbec1f7d0a8662faa33a212217c9e"
server_key_version: "encryptionkey@0"
I20260928 21:16:36.005564 22569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "3d4badc0d65b47c69ab5d1e95df1f643"
format_stamp: "Formatted at 2026-09-28 21:16:35 on dist-test-slave-v7xz"
server_key: "6fdcbd990c5def9c703d0fff5140aea6"
server_key_iv: "724dbec1f7d0a8662faa33a212217c9e"
server_key_version: "encryptionkey@0"
I20260928 21:16:36.009141 22569 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260928 21:16:36.011364 22584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:36.012538 22569 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:36.012687 22569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "3d4badc0d65b47c69ab5d1e95df1f643"
format_stamp: "Formatted at 2026-09-28 21:16:35 on dist-test-slave-v7xz"
server_key: "6fdcbd990c5def9c703d0fff5140aea6"
server_key_iv: "724dbec1f7d0a8662faa33a212217c9e"
server_key_version: "encryptionkey@0"
I20260928 21:16:36.012789 22569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:36.017383 22569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:36.020211 22569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:36.020406 22569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:36.027657 22636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33889 every 8 connection(s)
I20260928 21:16:36.027666 22569 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33889
I20260928 21:16:36.028698 22569 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:36.030121 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22569
I20260928 21:16:36.030293 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:36.030577 392 external_mini_cluster.cc:1428] Setting key 45f697b32677c5b65a1725d57b6a848c
I20260928 21:16:36.031672 22637 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:36.035959 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:36.032090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40695 (local address 127.0.98.62:33889)
0928 21:16:36.032488 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:36.032506 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:36.032549 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:16:36.033246 (+ 697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:36.033266 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:36.033547 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:36.033821 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:36.034702 (+ 881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:36.034706 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:36.034718 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:36.034728 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:36.034779 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:36.034797 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:36.034916 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:36.035047 (+ 131us) server_negotiation.cc:299] Negotiation successful
0928 21:16:36.035261 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261,"thread_start_us":142,"threads_started":1}
I20260928 21:16:36.037989 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643: Bootstrap starting.
I20260928 21:16:36.038812 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:36.040766 22637 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:36.041688 22637 log.cc:824] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:36.042873 22640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:36.041121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45417 (local address 127.0.98.62:33889)
0928 21:16:36.041255 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:36.041260 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:36.041442 (+ 182us) server_negotiation.cc:407] Connection header received
0928 21:16:36.041540 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:36.041545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:36.041588 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:36.041664 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:36.042427 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:36.042431 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:36.042436 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0928 21:16:36.042450 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:36.042478 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0928 21:16:36.042490 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:36.042572 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:36.042693 (+ 121us) server_negotiation.cc:299] Negotiation successful
0928 21:16:36.042724 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260928 21:16:36.043828 22637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643: No bootstrap required, opened a new log
I20260928 21:16:36.047333 22637 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260928 21:16:36.047595 22637 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260928 21:16:36.047685 22637 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d4badc0d65b47c69ab5d1e95df1f643, State: Initialized, Role: FOLLOWER
I20260928 21:16:36.048174 22637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260928 21:16:36.048306 22637 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260928 21:16:36.048395 22637 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260928 21:16:36.048493 22637 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:36.049512 22637 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260928 21:16:36.049870 22637 leader_election.cc:302] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d4badc0d65b47c69ab5d1e95df1f643; no voters:
I20260928 21:16:36.050180 22637 leader_election.cc:288] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260928 21:16:36.050345 22647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:36.050617 22647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [term 1 LEADER]: Becoming Leader. State: Replica: 3d4badc0d65b47c69ab5d1e95df1f643, State: Running, Role: LEADER
I20260928 21:16:36.050977 22647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } }
I20260928 21:16:36.051468 22637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:36.052565 22649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d4badc0d65b47c69ab5d1e95df1f643. Latest consensus state: current_term: 1 leader_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } } }
I20260928 21:16:36.052695 22649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:36.053124 22656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:36.053135 22648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d4badc0d65b47c69ab5d1e95df1f643" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33889 } } }
I20260928 21:16:36.053333 22648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643 [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:36.056351 22656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:36.062050 22656 catalog_manager.cc:1347] Generated new cluster ID: f29ffac0f0ff4cd990d22728763d4d50
I20260928 21:16:36.062171 22656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:36.093441 22656 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:36.093828 22656 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:36.105285 22656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d4badc0d65b47c69ab5d1e95df1f643: Generated new TSK 0
I20260928 21:16:36.106036 22656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260928 21:16:36.117686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22569
2026-09-28T21:16:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1415 ms)
[----------] 18 tests from AuthTokenIssuingTest (25494 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:36 dist-test-slave-v7xz krb5kdc[22668](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:36 dist-test-slave-v7xz krb5kdc[22668](info): set up 2 sockets
Sep 28 21:16:36 dist-test-slave-v7xz krb5kdc[22668](info): commencing operation
krb5kdc: starting...
W20260928 21:16:37.233207 392 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:37.361833 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33795
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33795,127.0.98.61:34257,127.0.98.60:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:37.473801 22681 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:16:37.474092 22681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:37.474115 22681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:37.474215 22681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:37.478391 22681 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:37.478454 22681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:37.478474 22681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:37.478490 22681 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:37.478506 22681 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:37.483824 22681 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33795,127.0.98.61:34257,127.0.98.60:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:37.484929 22681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:37.485968 22681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:37.494570 22689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:37.494607 22686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:37.494765 22681 server_base.cc:1034] running on GCE node
W20260928 21:16:37.494606 22687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:37.495448 22681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:37.496577 22681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:37.497782 22681 hybrid_clock.cc:648] HybridClock initialized: now 1790630197497753 us; error 46 us; skew 500 ppm
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:37.500721 22681 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:37.501713 22681 webserver.cc:460] Webserver started at http://127.0.98.62:35929/ using document root <none> and password file <none>
I20260928 21:16:37.502321 22681 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:37.502425 22681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:37.502672 22681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:37.504703 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "6b4d88284cf7458ab041897e6c7f4dad"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "d10a41147beb0ab3c8b2fa31d851ada3"
server_key_iv: "ad8f7f3bb6e6366290c405ad9af90d40"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.505348 22681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "6b4d88284cf7458ab041897e6c7f4dad"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "d10a41147beb0ab3c8b2fa31d851ada3"
server_key_iv: "ad8f7f3bb6e6366290c405ad9af90d40"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.509102 22681 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260928 21:16:37.511437 22696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:37.512576 22681 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:37.512734 22681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "6b4d88284cf7458ab041897e6c7f4dad"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "d10a41147beb0ab3c8b2fa31d851ada3"
server_key_iv: "ad8f7f3bb6e6366290c405ad9af90d40"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.512884 22681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:37.542330 22681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:37.545830 22681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:37.546062 22681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:37.553184 22681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33795
I20260928 21:16:37.553495 22748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33795 every 8 connection(s)
I20260928 21:16:37.554199 22681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:37.556977 22749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:37.558569 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22681
I20260928 21:16:37.558708 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:37.558979 392 external_mini_cluster.cc:1428] Setting key fb206b3e51c12099e298d01bf27b8789
I20260928 21:16:37.561767 22749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:37.577503 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.564087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33795 (local address 127.0.0.1:56030)
0928 21:16:37.564558 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.564611 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:37.565457 (+ 846us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:37.566278 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:37.566282 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:37.566597 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:37.566935 (+ 338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.566949 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.568292 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.568295 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.569105 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.569115 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.569806 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.569832 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.569943 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.570581 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:37.570607 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:37.572287 (+ 1680us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:37.574407 (+ 2120us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.574420 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.574436 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.574754 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.575075 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.575080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.575083 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.575246 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.575917 (+ 671us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:37.575928 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:37.576126 (+ 198us) client_negotiation.cc:769] Sending connection context
0928 21:16:37.576334 (+ 208us) client_negotiation.cc:240] Negotiation successful
0928 21:16:37.576762 (+ 428us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"mutex_wait_us":99,"thread_start_us":138,"threads_started":1}
I20260928 21:16:37.577502 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.564089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56030 (local address 127.0.98.62:33795)
0928 21:16:37.564687 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:37.564698 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:37.564743 (+ 45us) server_negotiation.cc:407] Connection header received
0928 21:16:37.565794 (+ 1051us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:37.565802 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:37.566065 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:37.566278 (+ 213us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:37.567114 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.568155 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.569244 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.569661 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.569834 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.572448 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:37.572456 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:37.572471 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:37.572509 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:37.574268 (+ 1759us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.574860 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.574865 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.574866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.574963 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.575380 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.575383 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.575384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.575769 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:37.575897 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:37.576395 (+ 498us) server_negotiation.cc:299] Negotiation successful
0928 21:16:37.576644 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":462,"spinlock_wait_cycles":5376,"thread_start_us":170,"threads_started":1}
I20260928 21:16:37.579941 22749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:37.580790 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.580479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:50964)
0928 21:16:37.580618 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.580700 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
W20260928 21:16:37.581010 22698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
I20260928 21:16:37.581851 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.581661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:50978)
0928 21:16:37.581758 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.581800 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:16:37.581993 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 21:16:37.585991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34257
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33795,127.0.98.61:34257,127.0.98.60:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:16:37.631243 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } attempt: 1
I20260928 21:16:37.632161 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.631863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:50982)
0928 21:16:37.632022 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.632084 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260928 21:16:37.632930 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.632666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:50992)
0928 21:16:37.632775 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.632866 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:16:37.633057 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
W20260928 21:16:37.697898 22756 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:16:37.698238 22756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:37.698318 22756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:37.698400 22756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:16:37.701231 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } attempt: 2
I20260928 21:16:37.702126 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.701801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:51006)
0928 21:16:37.701973 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.702048 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 21:16:37.702770 22756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:37.702852 22756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:37.702898 22756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260928 21:16:37.702896 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.702622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:51020)
0928 21:16:37.702746 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.702801 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260928 21:16:37.702934 22756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:37.702971 22756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260928 21:16:37.703027 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:34257: Network error: Client connection negotiation failed: client connection to 127.0.98.61:34257: connect: Connection refused (error 111)
I20260928 21:16:37.708257 22756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33795,127.0.98.61:34257,127.0.98.60:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:34257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22756
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:37.709364 22756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:37.710415 22756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:37.719383 22762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:37.719379 22764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:37.719375 22761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:37.719707 22756 server_base.cc:1034] running on GCE node
I20260928 21:16:37.720320 22756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:37.721475 22756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:37.722675 22756 hybrid_clock.cc:648] HybridClock initialized: now 1790630197722661 us; error 24 us; skew 500 ppm
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:37.725482 22756 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 21:16:37.726426 22756 webserver.cc:460] Webserver started at http://127.0.98.61:38853/ using document root <none> and password file <none>
I20260928 21:16:37.727043 22756 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:37.727154 22756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:37.727416 22756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:37.729744 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data/instance:
uuid: "68de637973a0408c99f93dffdd63203b"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "e38524a051aedde1c43a381a80ba3e5a"
server_key_iv: "e5dcd08bd8711804fb3ab5a8e6e56a1e"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.730386 22756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal/instance:
uuid: "68de637973a0408c99f93dffdd63203b"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "e38524a051aedde1c43a381a80ba3e5a"
server_key_iv: "e5dcd08bd8711804fb3ab5a8e6e56a1e"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.734438 22756 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260928 21:16:37.737068 22771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:37.738349 22756 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260928 21:16:37.738507 22756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal
uuid: "68de637973a0408c99f93dffdd63203b"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "e38524a051aedde1c43a381a80ba3e5a"
server_key_iv: "e5dcd08bd8711804fb3ab5a8e6e56a1e"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.738627 22756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:37.753937 22756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:37.756779 22756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:37.757030 22756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:37.764067 22756 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:34257
I20260928 21:16:37.764358 22823 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:34257 every 8 connection(s)
I20260928 21:16:37.765116 22756 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
I20260928 21:16:37.767908 22824 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:37.772182 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22756
I20260928 21:16:37.771914 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:37.772320 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-1/wal/instance
I20260928 21:16:37.772619 392 external_mini_cluster.cc:1428] Setting key c9af0e8a7b84f7cbee101230aa901470
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:37.789762 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.774269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56040 (local address 127.0.98.62:33795)
0928 21:16:37.774420 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:37.774424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:37.775344 (+ 920us) server_negotiation.cc:407] Connection header received
0928 21:16:37.776695 (+ 1351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:37.776698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:37.776746 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:37.776879 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:37.778503 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.779037 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.780916 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.781160 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.781245 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.785627 (+ 4382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:37.785630 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:37.785632 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:37.785658 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:37.787407 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.787936 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.787940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.787944 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.788032 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.788410 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.788414 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.788416 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.788697 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:37.788823 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:37.789444 (+ 621us) server_negotiation.cc:299] Negotiation successful
0928 21:16:37.789586 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:16:37.790591 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.774693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33795 (local address 127.0.0.1:56040)
0928 21:16:37.775159 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.775213 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:37.776459 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:37.777057 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:37.777068 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:37.777591 (+ 523us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:37.778322 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.778345 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.779176 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.779180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.780721 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.780737 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.781334 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.781338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.781562 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.782209 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:37.782226 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:37.785415 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:37.787522 (+ 2107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.787527 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.787542 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.787803 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.788148 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.788152 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.788158 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.788297 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.788820 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:37.788851 (+ 31us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:37.789161 (+ 310us) client_negotiation.cc:769] Sending connection context
0928 21:16:37.789359 (+ 198us) client_negotiation.cc:240] Negotiation successful
0928 21:16:37.789704 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":138,"threads_started":1}
I20260928 21:16:37.791996 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:37.794186 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } attempt: 3
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 21:16:37.802284 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40121
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33795,127.0.98.61:34257,127.0.98.60:40121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:16:37.802992 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.792713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:51026)
0928 21:16:37.792879 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.792893 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:37.792965 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:37.793663 (+ 698us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:37.793666 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:37.793844 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:37.793917 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.793923 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.795306 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.795309 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.796000 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.796008 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.796657 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.796660 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.796750 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.797290 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:37.797301 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:37.798600 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:37.801058 (+ 2458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.801061 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.801063 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.801332 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.801702 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.801707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.801712 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.801797 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.802334 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:37.802339 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:37.802444 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:16:37.802530 (+ 86us) client_negotiation.cc:240] Negotiation successful
0928 21:16:37.802723 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260928 21:16:37.803668 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.792900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51026 (local address 127.0.98.61:34257)
0928 21:16:37.793235 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:37.793244 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:37.793263 (+ 19us) server_negotiation.cc:407] Connection header received
0928 21:16:37.793295 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:37.793299 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:37.793457 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:37.793688 (+ 231us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:37.794051 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.795203 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.796116 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.796543 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.796663 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.798927 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:37.798938 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:37.798954 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:37.798978 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:37.800941 (+ 1963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.801475 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.801482 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.801486 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.801599 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.801899 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.801903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.801905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.802218 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:37.802726 (+ 508us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:37.802835 (+ 109us) server_negotiation.cc:299] Negotiation successful
0928 21:16:37.802989 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":154,"threads_started":1}
I20260928 21:16:37.804683 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.794635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:51032)
0928 21:16:37.794750 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.794766 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:37.794850 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:37.795281 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:37.795285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:37.795463 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:37.795603 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.795611 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.796406 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.796410 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.797155 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.797169 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.798171 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.798176 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.798337 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.798891 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:37.798905 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:37.800267 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:37.803073 (+ 2806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.803076 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.803078 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.803330 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.803645 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:37.803647 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:37.803649 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:37.803745 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:37.804167 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:37.804170 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:37.804275 (+ 105us) client_negotiation.cc:769] Sending connection context
0928 21:16:37.804384 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:16:37.804522 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260928 21:16:37.804698 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.794711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51032 (local address 127.0.98.61:34257)
0928 21:16:37.795037 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:37.795044 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:37.795057 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:16:37.795088 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:37.795090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:37.795124 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:37.795214 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:37.795735 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.796256 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.797783 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.798028 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.798108 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.800402 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:37.800405 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:37.800408 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:37.800431 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:37.802985 (+ 2554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.803431 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.803436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.803438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.803539 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:37.803867 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:37.803869 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:37.803871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:37.804057 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:37.804175 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:37.804443 (+ 268us) server_negotiation.cc:299] Negotiation successful
0928 21:16:37.804552 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":119,"threads_started":1}
I20260928 21:16:37.805917 22749 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:37.806104 22824 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:37.806615 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.806392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36846)
0928 21:16:37.806490 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.806550 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260928 21:16:37.806749 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.806516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36848)
0928 21:16:37.806610 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.806689 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260928 21:16:37.807165 22772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
I20260928 21:16:37.807375 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.807127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36852)
0928 21:16:37.807273 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.807318 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260928 21:16:37.807490 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
I20260928 21:16:37.808169 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.807962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36860)
0928 21:16:37.808082 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.808123 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260928 21:16:37.808336 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
I20260928 21:16:37.838729 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } attempt: 1
I20260928 21:16:37.839596 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.839299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36864)
0928 21:16:37.839445 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.839520 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260928 21:16:37.840374 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.840156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36868)
0928 21:16:37.840269 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.840319 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260928 21:16:37.840499 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
I20260928 21:16:37.857560 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } attempt: 1
I20260928 21:16:37.858455 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.858108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36884)
0928 21:16:37.858285 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.858362 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 21:16:37.859226 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.858981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36898)
0928 21:16:37.859107 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.859164 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260928 21:16:37.859385 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
W20260928 21:16:37.914573 22831 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:16:37.914887 22831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:37.914911 22831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:37.915023 22831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260928 21:16:37.915669 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } attempt: 2
I20260928 21:16:37.916502 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.916225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36910)
0928 21:16:37.916359 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.916424 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260928 21:16:37.917347 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.917084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36920)
0928 21:16:37.917197 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.917244 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260928 21:16:37.917467 22749 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
W20260928 21:16:37.919569 22831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:37.919656 22831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:37.919703 22831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:37.919759 22831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:37.919798 22831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:37.925338 22831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33795,127.0.98.61:34257,127.0.98.60:40121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22831
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:37.926479 22831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:37.927613 22831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260928 21:16:37.927559 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } attempt: 2
I20260928 21:16:37.928424 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.928093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36924)
0928 21:16:37.928266 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.928335 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260928 21:16:37.929226 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.929016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36934)
0928 21:16:37.929118 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.929173 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260928 21:16:37.929399 22824 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40121: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40121: connect: Connection refused (error 111)
W20260928 21:16:37.936774 22838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:37.936781 22840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:37.936787 22837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:37.936915 22831 server_base.cc:1034] running on GCE node
I20260928 21:16:37.937664 22831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:37.938866 22831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:37.940078 22831 hybrid_clock.cc:648] HybridClock initialized: now 1790630197940061 us; error 38 us; skew 500 ppm
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:37.943032 22831 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 21:16:37.944022 22831 webserver.cc:460] Webserver started at http://127.0.98.60:35689/ using document root <none> and password file <none>
I20260928 21:16:37.944609 22831 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:37.944708 22831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:37.945008 22831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:37.947185 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data/instance:
uuid: "6739b525da504f3990e767ea1026a338"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "428ded01d7f5e6862dd787e500b9fb0b"
server_key_iv: "1ee8023da4ca215e6b8dad0d833f9364"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.947768 22831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal/instance:
uuid: "6739b525da504f3990e767ea1026a338"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "428ded01d7f5e6862dd787e500b9fb0b"
server_key_iv: "1ee8023da4ca215e6b8dad0d833f9364"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.951601 22831 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260928 21:16:37.954180 22847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:37.955446 22831 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:37.955593 22831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal
uuid: "6739b525da504f3990e767ea1026a338"
format_stamp: "Formatted at 2026-09-28 21:16:37 on dist-test-slave-v7xz"
server_key: "428ded01d7f5e6862dd787e500b9fb0b"
server_key_iv: "1ee8023da4ca215e6b8dad0d833f9364"
server_key_version: "encryptionkey@0"
I20260928 21:16:37.955710 22831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:37.973771 22831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:37.976666 22831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:37.976938 22831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:37.983808 22831 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40121
I20260928 21:16:37.984098 22899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40121 every 8 connection(s)
I20260928 21:16:37.984845 22831 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
I20260928 21:16:37.987643 22900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:37.989724 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 22831
I20260928 21:16:37.989858 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790629916120794-392-0/minicluster-data/master-2/wal/instance
I20260928 21:16:37.990170 392 external_mini_cluster.cc:1428] Setting key 68a7c72bfddfccac07fdadcf2a93d121
I20260928 21:16:37.992141 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:16:37 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:38.005430 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.995258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.0.98.62:33795)
0928 21:16:37.995396 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:37.995401 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:37.995416 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:16:37.995461 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:37.995464 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:37.995513 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:37.995597 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:37.996403 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.997182 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.998392 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.998671 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.998817 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.001598 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.001603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.001608 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.001642 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.003404 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.003935 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.003939 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.003941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.004028 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.004354 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.004357 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.004359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.004577 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.004683 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.005130 (+ 447us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.005256 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:16:38.006242 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.991603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36938 (local address 127.0.98.60:40121)
0928 21:16:37.992042 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:37.992059 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:37.992097 (+ 38us) server_negotiation.cc:407] Connection header received
0928 21:16:37.993008 (+ 911us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:37.993030 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:37.993454 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:37.993766 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:37.994654 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.996064 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.997328 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.998064 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.998219 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.000959 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.000967 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.000993 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.001015 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.003215 (+ 2200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.003733 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.003737 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.003739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.003841 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.004182 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.004185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.004186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.004663 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.004821 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.005130 (+ 309us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.005358 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":276,"thread_start_us":150,"threads_started":1}
I20260928 21:16:38.006240 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:37.994632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33795 (local address 127.0.0.1:56050)
0928 21:16:37.995127 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:37.995177 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:37.995280 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:37.995670 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:37.995676 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:37.996099 (+ 423us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:37.996261 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.996267 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.997322 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.997325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.998257 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:37.998265 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:37.998862 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:37.998865 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:37.998946 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:37.999463 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:37.999498 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:38.001416 (+ 1918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:38.003532 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.003542 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.003555 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.003830 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.004127 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.004131 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.004136 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.004257 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.004683 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:38.004688 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:38.004883 (+ 195us) client_negotiation.cc:769] Sending connection context
0928 21:16:38.005057 (+ 174us) client_negotiation.cc:240] Negotiation successful
0928 21:16:38.005462 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":146,"thread_start_us":220,"threads_started":1}
I20260928 21:16:38.007462 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:16:38 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 28 21:16:38 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:38.016486 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.007978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:34257 (local address 127.0.0.1:51046)
0928 21:16:38.008125 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:38.008140 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:38.008242 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:38.008502 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:38.008505 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:38.008675 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:38.008776 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.008782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.009655 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.009657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.010204 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.010213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.010677 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.010681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.010763 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.011248 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:38.011259 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:38.012588 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:38.014863 (+ 2275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.014866 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.014867 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.015056 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.015369 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.015372 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.015374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.015455 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.015937 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:38.015940 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:38.016019 (+ 79us) client_negotiation.cc:769] Sending connection context
0928 21:16:38.016169 (+ 150us) client_negotiation.cc:240] Negotiation successful
0928 21:16:38.016300 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260928 21:16:38.016494 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.008076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51046 (local address 127.0.98.61:34257)
0928 21:16:38.008232 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.008238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.008256 (+ 18us) server_negotiation.cc:407] Connection header received
0928 21:16:38.008330 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.008332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.008379 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.008451 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.008906 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.009537 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.010320 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.010567 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.010649 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.012727 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.012730 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.012733 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.012764 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.014568 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.015168 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.015173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.015174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.015260 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.015558 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.015562 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.015564 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.015745 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.015875 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.016224 (+ 349us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.016338 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260928 21:16:38.016640 22749 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } attempt: 3
I20260928 21:16:38.017450 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.008654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56058 (local address 127.0.98.62:33795)
0928 21:16:38.008771 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.008774 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.008785 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:16:38.008885 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.008889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.008928 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.009011 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.009551 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.010058 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.011158 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.011383 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.011461 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.013986 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.013990 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.013992 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.014016 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.015404 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.015891 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.015896 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.015898 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.015987 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.016320 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.016323 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.016325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.016584 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.016936 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.017016 (+ 80us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.017140 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260928 21:16:38.018005 22900 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:38.020579 22824 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } attempt: 3
Sep 28 21:16:38 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:16:38 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:16:38 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 21:16:38.026648 22752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.017010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36946)
0928 21:16:38.017281 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:38.017292 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:38.017385 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:38.017647 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:38.017650 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:38.017830 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:38.017930 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.017938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.018946 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.018949 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.019830 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.019840 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.020385 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.020389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.020481 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.021000 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:38.021014 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:38.022323 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:38.024513 (+ 2190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.024517 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.024519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.024817 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.025183 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.025186 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.025189 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.025285 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.025719 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:38.025722 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:38.025811 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 21:16:38.025892 (+ 81us) client_negotiation.cc:240] Negotiation successful
0928 21:16:38.026483 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200}
I20260928 21:16:38.027460 22903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.017296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36946 (local address 127.0.98.60:40121)
0928 21:16:38.017431 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.017436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.017450 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:38.017483 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.017489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.017529 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.018048 (+ 519us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.018133 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.018817 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.019991 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.020261 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.020374 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.022466 (+ 2092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.022471 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.022476 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.022509 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.024349 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.024988 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.024992 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.024994 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.025077 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.025401 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.025404 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.025406 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.025609 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.027103 (+ 1494us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.027182 (+ 79us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.027290 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":60}
Sep 28 21:16:38 dist-test-slave-v7xz krb5kdc[22668](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630197, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 21:16:38.030807 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.018813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51060 (local address 127.0.98.61:34257)
0928 21:16:38.018925 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.018930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.019108 (+ 178us) server_negotiation.cc:407] Connection header received
0928 21:16:38.019195 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.019198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.019248 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.019333 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.020053 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.020933 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.022151 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.022497 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.022592 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.025734 (+ 3142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.025753 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.025756 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.025781 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.028427 (+ 2646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.029021 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.029025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.029027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.029118 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.029492 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.029502 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.029507 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.029793 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.030014 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.030474 (+ 460us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.030608 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260928 21:16:38.030915 22904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.018503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36948)
0928 21:16:38.018592 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:38.018606 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:38.018677 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:38.019190 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:38.019193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:38.019383 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:38.019469 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.019476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.021075 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.021078 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.021943 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.021953 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.022576 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.022581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.022679 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.023208 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:38.023223 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:38.025572 (+ 2349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:38.029378 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.029381 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.029383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.029615 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.029897 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.029900 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.029902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.029975 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.030418 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:38.030421 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:38.030511 (+ 90us) client_negotiation.cc:769] Sending connection context
0928 21:16:38.030588 (+ 77us) client_negotiation.cc:240] Negotiation successful
0928 21:16:38.030737 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260928 21:16:38.031100 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.018647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36948 (local address 127.0.98.60:40121)
0928 21:16:38.018957 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.018964 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.018974 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:16:38.019001 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.019003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.019034 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.019560 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.019613 (+ 53us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.020959 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.022092 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.022453 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.022849 (+ 396us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.025722 (+ 2873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.025726 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.025728 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.025752 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.029269 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.029730 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.029733 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.029735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.029809 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.030071 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.030077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.030082 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.030315 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.030740 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.030802 (+ 62us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.030919 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260928 21:16:38.031289 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad: Bootstrap starting.
I20260928 21:16:38.033636 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.020945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40121 (local address 127.0.0.1:36960)
0928 21:16:38.021191 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:38.021204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:38.021274 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:38.023588 (+ 2314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:38.023595 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:38.023779 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:38.023902 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.023909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.025025 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.025030 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.025831 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:38.025840 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.026511 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.026514 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:38.026602 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.027118 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:38.027133 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:38.028858 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:38.031835 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.031839 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.031841 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.032114 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.032497 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:38.032501 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:38.032503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:38.032598 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:38.033108 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:38.033111 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:38.033210 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 21:16:38.033313 (+ 103us) client_negotiation.cc:240] Negotiation successful
0928 21:16:38.033474 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20260928 21:16:38.033975 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: Bootstrap starting.
I20260928 21:16:38.034211 22906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.021406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36960 (local address 127.0.98.60:40121)
0928 21:16:38.022579 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.022588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.022600 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:38.022632 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.022634 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.022674 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.024030 (+ 1356us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.024165 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.024898 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.025954 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.026371 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.026822 (+ 451us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.029023 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.029028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.029030 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.029055 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.031691 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.032283 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.032286 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.032288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.032381 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.032727 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.032730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.032732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.032984 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.033828 (+ 844us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.033905 (+ 77us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.034033 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1095,"thread_start_us":105,"threads_started":1}
I20260928 21:16:38.036525 22900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:38.037174 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: Bootstrap starting.
I20260928 21:16:38.037467 22900 log.cc:824] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:38.039623 22824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:38.039681 22900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: No bootstrap required, opened a new log
I20260928 21:16:38.040441 22824 log.cc:824] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:38.041165 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260928 21:16:38.041404 22749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:38.042223 22749 log.cc:824] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:38.043707 22824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: No bootstrap required, opened a new log
I20260928 21:16:38.042531 22723 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6b4d88284cf7458ab041897e6c7f4dad"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56058
I20260928 21:16:38.044202 22749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad: No bootstrap required, opened a new log
I20260928 21:16:38.044600 22900 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.044891 22900 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6739b525da504f3990e767ea1026a338, State: Initialized, Role: FOLLOWER
I20260928 21:16:38.045228 22900 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.045811 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.046146 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.046622 22900 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:38.047075 22749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.047314 22749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b4d88284cf7458ab041897e6c7f4dad, State: Initialized, Role: FOLLOWER
I20260928 21:16:38.047676 22749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.047897 22824 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.048112 22824 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68de637973a0408c99f93dffdd63203b, State: Initialized, Role: FOLLOWER
I20260928 21:16:38.048409 22824 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.048440 22916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.048755 22916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.049208 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.049664 22749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:38.050037 22723 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:16:38.050140 22723 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:38.051400 22723 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
W20260928 21:16:38.052187 22922 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:38.052311 22922 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:38.049664 22824 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:38.052318 22723 leader_election.cc:288] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [CANDIDATE]: Term 1 election: Requested vote from peers 68de637973a0408c99f93dffdd63203b (127.0.98.61:34257), 6739b525da504f3990e767ea1026a338 (127.0.98.60:40121)
I20260928 21:16:38.050356 22831 master_runner.cc:186] Error getting master registration for 127.0.98.62:33795: OK, instance_id { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" instance_seqno: 1790630197551098 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:16:38.053841 22831 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:16:38.054293 22797 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b4d88284cf7458ab041897e6c7f4dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "68de637973a0408c99f93dffdd63203b"
I20260928 21:16:38.054522 22797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:38.055043 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.055990 22797 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4d88284cf7458ab041897e6c7f4dad in term 1.
I20260928 21:16:38.056097 22874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b4d88284cf7458ab041897e6c7f4dad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6739b525da504f3990e767ea1026a338"
I20260928 21:16:38.056308 22874 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:38.056402 22698 leader_election.cc:302] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68de637973a0408c99f93dffdd63203b, 6b4d88284cf7458ab041897e6c7f4dad; no voters:
I20260928 21:16:38.056694 22916 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:38.056936 22916 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [term 1 LEADER]: Becoming Leader. State: Replica: 6b4d88284cf7458ab041897e6c7f4dad, State: Running, Role: LEADER
I20260928 21:16:38.057261 22916 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } }
I20260928 21:16:38.057643 22874 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b4d88284cf7458ab041897e6c7f4dad in term 1.
I20260928 21:16:38.059132 22916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4d88284cf7458ab041897e6c7f4dad. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.055202 22681 master_runner.cc:186] Error getting master registration for 127.0.98.61:34257: OK, instance_id { permanent_uuid: "68de637973a0408c99f93dffdd63203b" instance_seqno: 1790630197762131 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:16:38.059269 22916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:38.059348 22681 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260928 21:16:38.063587 22953 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:38.063678 22953 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260928 21:16:38.066493 22935 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:38.066632 22935 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:38.067063 22952 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260928 21:16:38.067773 22952 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:38.077657 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.061734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.0.98.61:34257)
0928 21:16:38.061845 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.061851 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.065528 (+ 3677us) server_negotiation.cc:407] Connection header received
0928 21:16:38.065695 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.065698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.065744 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.065797 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.068869 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.069604 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.070786 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.071119 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.071192 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.072875 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.072879 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.072881 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.072908 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.074769 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.076188 (+ 1419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.076191 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.076193 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.076283 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.076594 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.076598 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.076600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.076862 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.076997 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.077368 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.077492 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260928 21:16:38.080724 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.061266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36970 (local address 127.0.98.60:40121)
0928 21:16:38.061383 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.061389 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.063999 (+ 2610us) server_negotiation.cc:407] Connection header received
0928 21:16:38.064196 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.064200 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.064253 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.064305 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.066757 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.067599 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.068860 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.069264 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.069342 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.073924 (+ 4582us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.073928 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.073931 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.073960 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.075653 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.079341 (+ 3688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.079345 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.079347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.079442 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.079751 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.079755 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.079757 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.079977 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.080107 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.080428 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.080558 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260928 21:16:38.082506 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:38.063061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56072 (local address 127.0.98.62:33795)
0928 21:16:38.063172 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.063176 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.069836 (+ 6660us) server_negotiation.cc:407] Connection header received
0928 21:16:38.070010 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.070013 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.070062 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.070115 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.070970 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.071665 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.072861 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:38.073216 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:38.073309 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:38.078482 (+ 5173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:38.078486 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:38.078489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:38.078519 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:38.080308 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.081088 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.081092 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.081094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.081185 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:38.081509 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:38.081513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:38.081515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:38.081728 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:38.081849 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:38.082220 (+ 371us) server_negotiation.cc:299] Negotiation successful
0928 21:16:38.082339 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260928 21:16:38.089759 22874 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [term 1 FOLLOWER]: Refusing update from remote peer 6b4d88284cf7458ab041897e6c7f4dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:38.089896 22797 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [term 1 FOLLOWER]: Refusing update from remote peer 6b4d88284cf7458ab041897e6c7f4dad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:38.090502 22946 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:38.091004 22916 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [LEADER]: Connected to new peer: Peer: permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:38.095469 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4d88284cf7458ab041897e6c7f4dad. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.095619 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.096043 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6b4d88284cf7458ab041897e6c7f4dad. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.096177 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.099424 22916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.100693 22910 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.100809 22910 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.101178 22916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:38.101341 22959 mvcc.cc:205] Tried to move back new op lower bound from 7334421291367223296 to 7334421291246198784. Current Snapshot: MvccSnapshot[applied={T|T < 7334421291367223296}]
I20260928 21:16:38.107692 22961 mvcc.cc:205] Tried to move back new op lower bound from 7334421291367223296 to 7334421291246198784. Current Snapshot: MvccSnapshot[applied={T|T < 7334421291367223296}]
I20260928 21:16:38.109442 22952 catalog_manager.cc:1347] Generated new cluster ID: 81f833d2b30e459f86cc8b4e686109ed
I20260928 21:16:38.109531 22952 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:38.119930 22916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.120088 22916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:38.121327 22923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6b4d88284cf7458ab041897e6c7f4dad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6b4d88284cf7458ab041897e6c7f4dad" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33795 } } peers { permanent_uuid: "68de637973a0408c99f93dffdd63203b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 34257 } } peers { permanent_uuid: "6739b525da504f3990e767ea1026a338" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40121 } } }
I20260928 21:16:38.121464 22923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:38.121779 22960 mvcc.cc:205] Tried to move back new op lower bound from 7334421291367223296 to 7334421291246198784. Current Snapshot: MvccSnapshot[applied={T|T < 7334421291367223296}]
I20260928 21:16:38.128719 22952 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:38.133186 22952 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:38.147840 22952 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6b4d88284cf7458ab041897e6c7f4dad: Generated new TSK 0
I20260928 21:16:38.148849 22952 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:16:38.183465 22753 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.180082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.98.62:33795)
0928 21:16:38.180215 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.180220 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.182286 (+ 2066us) server_negotiation.cc:407] Connection header received
0928 21:16:38.182504 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.182508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.182562 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.182616 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.183301 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56080: BlockingRecv error: recv got EOF from 127.0.0.1:56080 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 21:16:38.184985 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.180941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51080 (local address 127.0.98.61:34257)
0928 21:16:38.181063 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.181067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.183971 (+ 2904us) server_negotiation.cc:407] Connection header received
0928 21:16:38.184154 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.184157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.184209 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.184260 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.184799 (+ 539us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51080: BlockingRecv error: recv got EOF from 127.0.0.1:51080 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260928 21:16:38.186589 22905 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.181677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36972 (local address 127.0.98.60:40121)
0928 21:16:38.181791 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.181795 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.185552 (+ 3757us) server_negotiation.cc:407] Connection header received
0928 21:16:38.185781 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.185785 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.185851 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.185903 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.186440 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36972: BlockingRecv error: recv got EOF from 127.0.0.1:36972 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260928 21:16:38.191924 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.190179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51088 (local address 127.0.98.61:34257)
0928 21:16:38.190315 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.190320 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.190827 (+ 507us) server_negotiation.cc:407] Connection header received
0928 21:16:38.191074 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.191078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.191161 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.191234 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.191807 (+ 573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51088: BlockingRecv error: recv got EOF from 127.0.0.1:51088 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 21:16:38.197570 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.195833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.0.98.61:34257)
0928 21:16:38.195949 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.195954 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.196546 (+ 592us) server_negotiation.cc:407] Connection header received
0928 21:16:38.196749 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.196753 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.196807 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.196893 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.197450 (+ 557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51104: BlockingRecv error: recv got EOF from 127.0.0.1:51104 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260928 21:16:38.203922 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.202188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.0.98.61:34257)
0928 21:16:38.202323 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.202328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.202887 (+ 559us) server_negotiation.cc:407] Connection header received
0928 21:16:38.203102 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.203106 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.203162 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.203231 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.203799 (+ 568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51108: BlockingRecv error: recv got EOF from 127.0.0.1:51108 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 21:16:38.212323 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.210486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.0.98.61:34257)
0928 21:16:38.210643 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.210648 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.211259 (+ 611us) server_negotiation.cc:407] Connection header received
0928 21:16:38.211483 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.211487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.211543 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.211601 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.212190 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51120: BlockingRecv error: recv got EOF from 127.0.0.1:51120 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260928 21:16:38.224939 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.222983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51122 (local address 127.0.98.61:34257)
0928 21:16:38.223135 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.223140 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.223808 (+ 668us) server_negotiation.cc:407] Connection header received
0928 21:16:38.224037 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.224042 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.224107 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.224168 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.224784 (+ 616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51122: BlockingRecv error: recv got EOF from 127.0.0.1:51122 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260928 21:16:38.246017 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.244003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51134 (local address 127.0.98.61:34257)
0928 21:16:38.244139 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.244144 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.244807 (+ 663us) server_negotiation.cc:407] Connection header received
0928 21:16:38.245086 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.245090 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.245155 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.245233 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.245886 (+ 653us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51134: BlockingRecv error: recv got EOF from 127.0.0.1:51134 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 21:16:38.282105 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.280669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51138 (local address 127.0.98.61:34257)
0928 21:16:38.280802 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.280806 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.281129 (+ 323us) server_negotiation.cc:407] Connection header received
0928 21:16:38.281313 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.281316 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.281374 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.281477 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.281979 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51138: BlockingRecv error: recv got EOF from 127.0.0.1:51138 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 21:16:38.349773 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.348507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.0.98.61:34257)
0928 21:16:38.348654 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.348659 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.348798 (+ 139us) server_negotiation.cc:407] Connection header received
0928 21:16:38.348945 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.348948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.348999 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.349081 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.349645 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51140: BlockingRecv error: recv got EOF from 127.0.0.1:51140 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260928 21:16:38.481112 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.480067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.0.98.61:34257)
0928 21:16:38.480237 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.480241 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.480261 (+ 20us) server_negotiation.cc:407] Connection header received
0928 21:16:38.480378 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.480380 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.480430 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.480523 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.481006 (+ 483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51152: BlockingRecv error: recv got EOF from 127.0.0.1:51152 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260928 21:16:38.740598 22830 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:38.739449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51166 (local address 127.0.98.61:34257)
0928 21:16:38.739593 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:38.739600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:38.739699 (+ 99us) server_negotiation.cc:407] Connection header received
0928 21:16:38.739888 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:38.739891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:38.739941 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:38.740038 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:38.740486 (+ 448us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51166: BlockingRecv error: recv got EOF from 127.0.0.1:51166 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20260928 21:16:39.053757 22922 catalog_manager.cc:1259] Loaded cluster ID: 81f833d2b30e459f86cc8b4e686109ed
I20260928 21:16:39.053841 22922 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: loading cluster ID for follower catalog manager: success
I20260928 21:16:39.055680 22922 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: acquiring CA information for follower catalog manager: success
I20260928 21:16:39.056972 22922 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6739b525da504f3990e767ea1026a338: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:16:39.065076 22953 catalog_manager.cc:1259] Loaded cluster ID: 81f833d2b30e459f86cc8b4e686109ed
I20260928 21:16:39.065152 22953 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: loading cluster ID for follower catalog manager: success
I20260928 21:16:39.066797 22953 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: acquiring CA information for follower catalog manager: success
I20260928 21:16:39.068027 22953 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 68de637973a0408c99f93dffdd63203b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:16:39.259007 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:39.255097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51180 (local address 127.0.98.61:34257)
0928 21:16:39.255341 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:39.255345 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:39.255447 (+ 102us) server_negotiation.cc:407] Connection header received
0928 21:16:39.255553 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:39.255556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:39.255681 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:39.255791 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:39.256030 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.256647 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.257539 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.257764 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.257860 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:39.258142 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:39.258594 (+ 452us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:39.258690 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:39.258766 (+ 76us) server_negotiation.cc:299] Negotiation successful
0928 21:16:39.258855 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":92,"threads_started":1}
I20260928 21:16:39.266201 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:39.262379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36976 (local address 127.0.98.60:40121)
0928 21:16:39.262667 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:39.262673 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:39.262687 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:39.262773 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:39.262775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:39.262896 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:39.263035 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:39.263265 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.263779 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.264652 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.264916 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.265036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:39.265361 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:39.265790 (+ 429us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:39.265941 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:39.265982 (+ 41us) server_negotiation.cc:299] Negotiation successful
0928 21:16:39.266038 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":89,"threads_started":1}
I20260928 21:16:39.273480 23035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:39.269649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51196 (local address 127.0.98.61:34257)
0928 21:16:39.269781 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:39.269784 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:39.270004 (+ 220us) server_negotiation.cc:407] Connection header received
0928 21:16:39.270177 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:39.270181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:39.270291 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:39.270367 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:39.270716 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.271480 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.272320 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.272603 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.272705 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:39.272980 (+ 275us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:39.273075 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:39.273145 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:39.273282 (+ 137us) server_negotiation.cc:299] Negotiation successful
0928 21:16:39.273336 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260928 21:16:39.274010 23041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:39.269513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36986 (local address 127.0.98.60:40121)
0928 21:16:39.269660 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:39.269666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:39.269941 (+ 275us) server_negotiation.cc:407] Connection header received
0928 21:16:39.270107 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:39.270110 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:39.270156 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:39.270236 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:39.270525 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.271294 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.272622 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.272989 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.273092 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:39.273384 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:39.273463 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:39.273514 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:39.273837 (+ 323us) server_negotiation.cc:299] Negotiation successful
0928 21:16:39.273895 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260928 21:16:39.274935 23048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:39.269476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56096 (local address 127.0.98.62:33795)
0928 21:16:39.269947 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:39.269951 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:39.269966 (+ 15us) server_negotiation.cc:407] Connection header received
0928 21:16:39.270060 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:39.270064 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:39.270244 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:39.270372 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:39.270632 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.271198 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.273178 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:39.273526 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:39.273636 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:39.274090 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:39.274467 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:39.274558 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:39.274718 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:16:39.274787 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":61,"threads_started":1}
I20260928 21:16:39.286830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22681
I20260928 21:16:39.293514 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22756
I20260928 21:16:39.300422 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 22831
2026-09-28T21:16:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3184 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 28 21:16:39 dist-test-slave-v7xz krb5kdc[23052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 28 21:16:39 dist-test-slave-v7xz krb5kdc[23052](info): set up 2 sockets
Sep 28 21:16:39 dist-test-slave-v7xz krb5kdc[23052](info): commencing operation
krb5kdc: starting...
W20260928 21:16:40.345364 392 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/test-user.keytab.
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-28T21:16:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-28T21:16:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260928 21:16:40.471467 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44565
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44565,127.0.98.61:32825,127.0.98.60:45593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260928 21:16:40.584722 23065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:16:40.585160 23065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:40.585189 23065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:40.585245 23065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:40.589458 23065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:40.589524 23065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:40.589543 23065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:40.589560 23065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:40.589576 23065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:40.595042 23065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44565,127.0.98.61:32825,127.0.98.60:45593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:40.596153 23065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:40.597245 23065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:40.605984 23070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:40.605974 23071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:40.605974 23073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:40.606364 23065 server_base.cc:1034] running on GCE node
I20260928 21:16:40.607012 23065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:40.608100 23065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:40.609295 23065 hybrid_clock.cc:648] HybridClock initialized: now 1790630200609273 us; error 34 us; skew 500 ppm
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:40.612071 23065 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260928 21:16:40.613024 23065 webserver.cc:460] Webserver started at http://127.0.98.62:41289/ using document root <none> and password file <none>
I20260928 21:16:40.613778 23065 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:40.613879 23065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:40.614152 23065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:40.616305 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data/instance:
uuid: "91b36825dbaa44769aa2e0adca43097e"
format_stamp: "Formatted at 2026-09-28 21:16:40 on dist-test-slave-v7xz"
server_key: "b8822796aeb6a8801f24fe5a5fa71a61"
server_key_iv: "da044f930db6592c5d136e806120afa3"
server_key_version: "encryptionkey@0"
I20260928 21:16:40.616917 23065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal/instance:
uuid: "91b36825dbaa44769aa2e0adca43097e"
format_stamp: "Formatted at 2026-09-28 21:16:40 on dist-test-slave-v7xz"
server_key: "b8822796aeb6a8801f24fe5a5fa71a61"
server_key_iv: "da044f930db6592c5d136e806120afa3"
server_key_version: "encryptionkey@0"
I20260928 21:16:40.620568 23065 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260928 21:16:40.623157 23080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:40.624431 23065 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260928 21:16:40.624572 23065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal
uuid: "91b36825dbaa44769aa2e0adca43097e"
format_stamp: "Formatted at 2026-09-28 21:16:40 on dist-test-slave-v7xz"
server_key: "b8822796aeb6a8801f24fe5a5fa71a61"
server_key_iv: "da044f930db6592c5d136e806120afa3"
server_key_version: "encryptionkey@0"
I20260928 21:16:40.624704 23065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:40.637063 23065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:40.639964 23065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:40.640187 23065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:40.647140 23065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44565
I20260928 21:16:40.647444 23132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44565 every 8 connection(s)
I20260928 21:16:40.648150 23065 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/data/info.pb
I20260928 21:16:40.651046 23133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:40.654883 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:40.657852 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 23065
I20260928 21:16:40.657976 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-0/wal/instance
I20260928 21:16:40.658337 392 external_mini_cluster.cc:1428] Setting key 92a80dbc849c82aa350ed470758d304b
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260928 21:16:40.670331 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.657191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44565 (local address 127.0.0.1:57550)
0928 21:16:40.657700 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.657746 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:40.658736 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:40.659430 (+ 694us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:40.659435 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:40.659887 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:40.660261 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.660275 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.661531 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.661534 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.662256 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.662265 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.662916 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.662920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.663070 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.663738 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:40.663753 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:40.665193 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:40.667163 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.667172 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.667189 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.667409 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.667814 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.667817 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.667819 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.667935 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.668626 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:40.668632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:40.668766 (+ 134us) client_negotiation.cc:769] Sending connection context
0928 21:16:40.668992 (+ 226us) client_negotiation.cc:240] Negotiation successful
0928 21:16:40.669359 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":155,"thread_start_us":125,"threads_started":1}
I20260928 21:16:40.670331 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.658120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57550 (local address 127.0.98.62:44565)
0928 21:16:40.658420 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:40.658431 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:40.658474 (+ 43us) server_negotiation.cc:407] Connection header received
0928 21:16:40.659045 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:40.659050 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:40.659242 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:40.659436 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:40.660426 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.661373 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.662382 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.662762 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.662945 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.665351 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:40.665356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:40.665366 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:40.665386 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:40.667054 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.667525 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.667530 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.667532 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.667674 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.668042 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.668047 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.668049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.668495 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:40.668643 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:40.669053 (+ 410us) server_negotiation.cc:299] Negotiation successful
0928 21:16:40.669283 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260928 21:16:40.673098 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:40.673940 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.673584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56598)
0928 21:16:40.673749 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.673844 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260928 21:16:40.674196 23081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
I20260928 21:16:40.675067 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.674796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56606)
0928 21:16:40.674932 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.674990 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260928 21:16:40.675256 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32825: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260928 21:16:40.684345 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32825
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44565,127.0.98.61:32825,127.0.98.60:45593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:16:40.724587 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } attempt: 1
I20260928 21:16:40.725559 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.725239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56618)
0928 21:16:40.725411 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.725478 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260928 21:16:40.726359 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.726160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56632)
0928 21:16:40.726264 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.726308 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:16:40.726572 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32825: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
I20260928 21:16:40.794759 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } attempt: 2
I20260928 21:16:40.795612 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.795311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56640)
0928 21:16:40.795463 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.795536 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 21:16:40.796401 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.796178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56650)
0928 21:16:40.796291 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.796337 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260928 21:16:40.796609 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:32825: Network error: Client connection negotiation failed: client connection to 127.0.98.61:32825: connect: Connection refused (error 111)
W20260928 21:16:40.798622 23140 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:16:40.798982 23140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:40.799037 23140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:40.799122 23140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:40.803783 23140 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:40.803853 23140 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:40.803874 23140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:40.803927 23140 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:40.803968 23140 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:40.810139 23140 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44565,127.0.98.61:32825,127.0.98.60:45593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:32825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23140
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:40.811281 23140 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:40.812410 23140 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:40.821720 23145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:40.821715 23148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:40.821717 23146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:40.822088 23140 server_base.cc:1034] running on GCE node
I20260928 21:16:40.822682 23140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:40.823807 23140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:40.824993 23140 hybrid_clock.cc:648] HybridClock initialized: now 1790630200824961 us; error 53 us; skew 500 ppm
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:40.828002 23140 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260928 21:16:40.829005 23140 webserver.cc:460] Webserver started at http://127.0.98.61:41517/ using document root <none> and password file <none>
I20260928 21:16:40.829568 23140 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:40.829639 23140 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:40.829877 23140 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:40.831878 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data/instance:
uuid: "6cb00c78961a4afdbefee5330d09c7a7"
format_stamp: "Formatted at 2026-09-28 21:16:40 on dist-test-slave-v7xz"
server_key: "757dbdc34d3fa9f5acf2e4b9788d4187"
server_key_iv: "59d76a4a15f4758d17f7e9f8601ff1ff"
server_key_version: "encryptionkey@0"
I20260928 21:16:40.832448 23140 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal/instance:
uuid: "6cb00c78961a4afdbefee5330d09c7a7"
format_stamp: "Formatted at 2026-09-28 21:16:40 on dist-test-slave-v7xz"
server_key: "757dbdc34d3fa9f5acf2e4b9788d4187"
server_key_iv: "59d76a4a15f4758d17f7e9f8601ff1ff"
server_key_version: "encryptionkey@0"
I20260928 21:16:40.836076 23140 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260928 21:16:40.838477 23155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:40.839620 23140 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:40.839749 23140 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal
uuid: "6cb00c78961a4afdbefee5330d09c7a7"
format_stamp: "Formatted at 2026-09-28 21:16:40 on dist-test-slave-v7xz"
server_key: "757dbdc34d3fa9f5acf2e4b9788d4187"
server_key_iv: "59d76a4a15f4758d17f7e9f8601ff1ff"
server_key_version: "encryptionkey@0"
I20260928 21:16:40.839851 23140 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:40.856799 23140 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:40.859716 23140 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:40.859936 23140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:40.867092 23140 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:32825
I20260928 21:16:40.867427 23207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:32825 every 8 connection(s)
I20260928 21:16:40.868079 23140 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/data/info.pb
I20260928 21:16:40.870853 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 23140
I20260928 21:16:40.870931 23208 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:40.870955 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-1/wal/instance
I20260928 21:16:40.871281 392 external_mini_cluster.cc:1428] Setting key 5f5797e9671583df86d8ce9352a76bad
I20260928 21:16:40.875481 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:40.887863 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } attempt: 3
I20260928 21:16:40.891211 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.877475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57552 (local address 127.0.98.62:44565)
0928 21:16:40.877641 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:40.877646 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:40.878508 (+ 862us) server_negotiation.cc:407] Connection header received
0928 21:16:40.879548 (+ 1040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:40.879551 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:40.879605 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:40.879693 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:40.881053 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.881804 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.883430 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.883858 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.883982 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.886965 (+ 2983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:40.886969 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:40.886971 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:40.886997 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:40.888746 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.889455 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.889461 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.889466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.889562 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.889943 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.889946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.889949 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.890191 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:40.890293 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:40.890858 (+ 565us) server_negotiation.cc:299] Negotiation successful
0928 21:16:40.891030 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260928 21:16:40.892096 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.877856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44565 (local address 127.0.0.1:57552)
0928 21:16:40.878330 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.878380 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:40.879335 (+ 955us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:40.879915 (+ 580us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:40.879922 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:40.880266 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:40.880914 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.880929 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.881942 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.881948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.883302 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.883312 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.884024 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.884027 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.884166 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.884790 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:40.884806 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:40.886770 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:40.888905 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.888913 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.888934 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.889332 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.889679 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.889683 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.889685 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.889825 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.890327 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:40.890335 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:40.890592 (+ 257us) client_negotiation.cc:769] Sending connection context
0928 21:16:40.890746 (+ 154us) client_negotiation.cc:240] Negotiation successful
0928 21:16:40.891053 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":144,"threads_started":1}
I20260928 21:16:40.894271 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 21:16:40.898932 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.888407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56666)
0928 21:16:40.888584 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.888601 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:40.888693 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:40.889570 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:40.889576 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:40.889756 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:40.889886 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.889892 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.891116 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.891120 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.891890 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.891901 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.892662 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.892665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.892771 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.893337 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:40.893351 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:40.894742 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:40.897028 (+ 2286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.897031 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.897033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.897332 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.897698 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.897700 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.897702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.897782 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.898363 (+ 581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:40.898367 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:40.898492 (+ 125us) client_negotiation.cc:769] Sending connection context
0928 21:16:40.898640 (+ 148us) client_negotiation.cc:240] Negotiation successful
0928 21:16:40.898802 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260928 21:16:40.899416 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.888640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56666 (local address 127.0.98.61:32825)
0928 21:16:40.889107 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:40.889116 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:40.889137 (+ 21us) server_negotiation.cc:407] Connection header received
0928 21:16:40.889178 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:40.889182 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:40.889343 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:40.889555 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:40.889983 (+ 428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.890899 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.892014 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.892485 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.892617 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.894903 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:40.894913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:40.894927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:40.894954 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:40.896911 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.897451 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.897458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.897460 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.897550 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.897880 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.897887 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.897891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.898249 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:40.898358 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:40.898751 (+ 393us) server_negotiation.cc:299] Negotiation successful
0928 21:16:40.898890 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":141,"threads_started":1}
Sep 28 21:16:40 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260928 21:16:40.900893 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-task8iL2mC/build/debug/bin/kudu
/tmp/dist-test-task8iL2mC/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45593
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42791
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44565,127.0.98.61:32825,127.0.98.60:45593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260928 21:16:40.901573 23133 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:40.902371 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.902105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55006)
0928 21:16:40.902239 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.902294 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260928 21:16:40.903189 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.902885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55022)
0928 21:16:40.903050 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.903098 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260928 21:16:40.903345 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45593: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
I20260928 21:16:40.903717 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.894826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56682)
0928 21:16:40.894928 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.894943 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:40.895010 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:40.895346 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:40.895348 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:40.895558 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:40.895691 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.895704 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.896519 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.896523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.897303 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:40.897314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.897953 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.897956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:40.898072 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.898541 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:40.898553 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:40.900040 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:40.902081 (+ 2041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.902087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.902092 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.902375 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.902665 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:40.902668 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:40.902669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:40.902741 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:40.903232 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:40.903235 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:40.903347 (+ 112us) client_negotiation.cc:769] Sending connection context
0928 21:16:40.903456 (+ 109us) client_negotiation.cc:240] Negotiation successful
0928 21:16:40.903588 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260928 21:16:40.904048 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.894885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.0.98.61:32825)
0928 21:16:40.895149 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:40.895161 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:40.895173 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:40.895200 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:40.895201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:40.895234 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:40.895339 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:40.895840 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.896395 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.897443 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:40.897801 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:40.897921 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:40.900187 (+ 2266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:40.900191 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:40.900193 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:40.900216 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:40.901961 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.902496 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.902500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.902503 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.902589 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:40.902829 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:40.902834 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:40.902837 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:40.903107 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:40.903217 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:40.903538 (+ 321us) server_negotiation.cc:299] Negotiation successful
0928 21:16:40.903673 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":214,"thread_start_us":101,"threads_started":1}
I20260928 21:16:40.904693 23208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:40.905488 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.905190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55024)
0928 21:16:40.905302 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.905420 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260928 21:16:40.905757 23159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
I20260928 21:16:40.906703 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.906506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55036)
0928 21:16:40.906604 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.906645 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:16:40.906940 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45593: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
I20260928 21:16:40.934530 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } attempt: 1
I20260928 21:16:40.935431 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.935105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55052)
0928 21:16:40.935275 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.935348 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260928 21:16:40.936301 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.936069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55056)
0928 21:16:40.936197 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.936249 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260928 21:16:40.936461 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45593: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
I20260928 21:16:40.956276 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } attempt: 1
I20260928 21:16:40.957190 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.956883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55064)
0928 21:16:40.957036 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.957109 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260928 21:16:40.957969 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:40.957750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55072)
0928 21:16:40.957860 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:40.957908 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260928 21:16:40.958127 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45593: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
I20260928 21:16:41.011626 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } attempt: 2
I20260928 21:16:41.012516 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.012201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55084)
0928 21:16:41.012363 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.012430 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260928 21:16:41.013474 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.013200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55086)
0928 21:16:41.013327 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.013380 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260928 21:16:41.013614 23133 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45593: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
W20260928 21:16:41.015676 23215 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260928 21:16:41.016032 23215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260928 21:16:41.016101 23215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260928 21:16:41.016179 23215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260928 21:16:41.020474 23215 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260928 21:16:41.020586 23215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260928 21:16:41.020635 23215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260928 21:16:41.020677 23215 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260928 21:16:41.020712 23215 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260928 21:16:41.026271 23215 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42791
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44565,127.0.98.61:32825,127.0.98.60:45593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:45593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 28 Sep 2026 20:27:13 UTC on ccycloud.quasar-nicwgn.root.comops.site
build id 4678652
I20260928 21:16:41.026281 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } attempt: 2
I20260928 21:16:41.027179 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.026868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55090)
0928 21:16:41.027019 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.027089 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260928 21:16:41.027635 23215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260928 21:16:41.028033 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.027761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55094)
0928 21:16:41.027857 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.027955 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260928 21:16:41.028177 23208 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:45593: Network error: Client connection negotiation failed: client connection to 127.0.98.60:45593: connect: Connection refused (error 111)
I20260928 21:16:41.028898 23215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260928 21:16:41.037457 23224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260928 21:16:41.037515 23221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:41.037564 23215 server_base.cc:1034] running on GCE node
W20260928 21:16:41.037515 23222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260928 21:16:41.038335 23215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260928 21:16:41.039469 23215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260928 21:16:41.040666 23215 hybrid_clock.cc:648] HybridClock initialized: now 1790630201040646 us; error 33 us; skew 500 ppm
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260928 21:16:41.043497 23215 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260928 21:16:41.044440 23215 webserver.cc:460] Webserver started at http://127.0.98.60:33067/ using document root <none> and password file <none>
I20260928 21:16:41.045033 23215 fs_manager.cc:359] Metadata directory not provided
I20260928 21:16:41.045099 23215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260928 21:16:41.045275 23215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260928 21:16:41.047313 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data/instance:
uuid: "de6345f5ba4646c7bde2546e6d2fe84c"
format_stamp: "Formatted at 2026-09-28 21:16:41 on dist-test-slave-v7xz"
server_key: "85350bba5584e6711bfed5cb551ac54c"
server_key_iv: "bba6e3e2a18361fe76f75c2cfb90eeb5"
server_key_version: "encryptionkey@0"
I20260928 21:16:41.047919 23215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal/instance:
uuid: "de6345f5ba4646c7bde2546e6d2fe84c"
format_stamp: "Formatted at 2026-09-28 21:16:41 on dist-test-slave-v7xz"
server_key: "85350bba5584e6711bfed5cb551ac54c"
server_key_iv: "bba6e3e2a18361fe76f75c2cfb90eeb5"
server_key_version: "encryptionkey@0"
I20260928 21:16:41.051683 23215 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.003s
I20260928 21:16:41.054174 23231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260928 21:16:41.055421 23215 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260928 21:16:41.055581 23215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data,/tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal
uuid: "de6345f5ba4646c7bde2546e6d2fe84c"
format_stamp: "Formatted at 2026-09-28 21:16:41 on dist-test-slave-v7xz"
server_key: "85350bba5584e6711bfed5cb551ac54c"
server_key_iv: "bba6e3e2a18361fe76f75c2cfb90eeb5"
server_key_version: "encryptionkey@0"
I20260928 21:16:41.055706 23215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260928 21:16:41.076606 23215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260928 21:16:41.079720 23215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260928 21:16:41.079933 23215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260928 21:16:41.087180 23215 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:45593
I20260928 21:16:41.087472 23283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:45593 every 8 connection(s)
I20260928 21:16:41.088258 23215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/data/info.pb
I20260928 21:16:41.088593 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-task8iL2mC/build/debug/bin/kudu as pid 23215
I20260928 21:16:41.088728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task8iL2mC/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790629916120794-392-0/minicluster-data/master-2/wal/instance
I20260928 21:16:41.089092 392 external_mini_cluster.cc:1428] Setting key af1f21907faecc5b31d4ffe17f30ef66
I20260928 21:16:41.091569 23284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260928 21:16:41.096248 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260928 21:16:41.104789 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.090529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55096 (local address 127.0.98.60:45593)
0928 21:16:41.090950 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.090969 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.091011 (+ 42us) server_negotiation.cc:407] Connection header received
0928 21:16:41.091885 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.091908 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.092340 (+ 432us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.092712 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.093515 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.095045 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.095905 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.096575 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.096755 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.099941 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.099945 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.099954 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.099980 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.101734 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.102202 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.102211 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.102219 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.102374 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.102712 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.102717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.102719 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.103138 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.103354 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.103686 (+ 332us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.103962 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"spinlock_wait_cycles":896,"thread_start_us":109,"threads_started":1}
I20260928 21:16:41.108644 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.098374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57560 (local address 127.0.98.62:44565)
0928 21:16:41.098509 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.098513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.099018 (+ 505us) server_negotiation.cc:407] Connection header received
0928 21:16:41.099154 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.099156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.099203 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.099311 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.100186 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.101003 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.101912 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.102117 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.102245 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.104876 (+ 2631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.104880 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.104883 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.104923 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.106563 (+ 1640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.107107 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.107113 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.107115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.107205 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.107589 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.107593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.107595 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.107865 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.108013 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.108340 (+ 327us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.108519 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":54}
I20260928 21:16:41.108707 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.098315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44565 (local address 127.0.0.1:57560)
0928 21:16:41.098847 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.098907 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:41.099050 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:41.099402 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:41.099408 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:41.099910 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:41.100054 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.100060 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.101121 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.101124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.101743 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.101752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.102259 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.102261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.102343 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.103049 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:41.103063 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:41.104687 (+ 1624us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:41.106696 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.106707 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.106714 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.107010 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.107318 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.107323 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.107326 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.107455 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.108015 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:41.108020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:41.108119 (+ 99us) client_negotiation.cc:769] Sending connection context
0928 21:16:41.108251 (+ 132us) client_negotiation.cc:240] Negotiation successful
0928 21:16:41.108453 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":123,"threads_started":1}
I20260928 21:16:41.109877 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } has no permanent_uuid. Determining permanent_uuid...
I20260928 21:16:41.112766 23133 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } attempt: 3
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 21:16:41.118389 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.107708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57562 (local address 127.0.98.62:44565)
0928 21:16:41.108109 (+ 401us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.108117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.108127 (+ 10us) server_negotiation.cc:407] Connection header received
0928 21:16:41.108155 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.108157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.108192 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.108307 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.109119 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.109726 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.110864 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.111097 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.111180 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.114115 (+ 2935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.114119 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.114121 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.114153 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.115954 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.116454 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.116457 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.116459 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.116553 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.117007 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.117011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.117014 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.117249 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.117454 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.118151 (+ 697us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.118248 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":323,"thread_start_us":117,"threads_started":1}
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 21:16:41.119313 23208 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } attempt: 3
I20260928 21:16:41.121361 23214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.110526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56684 (local address 127.0.98.61:32825)
0928 21:16:41.110665 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.110670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.110686 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:16:41.110732 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.110735 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.110789 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.110877 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.111487 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.112153 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.113344 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.113716 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.113778 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.116619 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.116625 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.116628 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.116658 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.118556 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.119068 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.119077 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.119080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.119193 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.120358 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.120362 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.120364 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.120590 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.120692 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.121066 (+ 374us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.121198 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260928 21:16:41.122044 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.110421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:32825 (local address 127.0.0.1:56684)
0928 21:16:41.110538 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.110553 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:41.110631 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:41.110998 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:41.111001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:41.111184 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:41.111297 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.111304 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.112306 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.112311 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.113211 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.113221 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.113954 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.113960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.114057 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.114526 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:41.114539 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:41.116465 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:41.118696 (+ 2231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.118699 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.118701 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.118947 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.119313 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.119316 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.119318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.119414 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.120736 (+ 1322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:41.120742 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:41.120880 (+ 138us) client_negotiation.cc:769] Sending connection context
0928 21:16:41.121759 (+ 879us) client_negotiation.cc:240] Negotiation successful
0928 21:16:41.121892 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260928 21:16:41.122965 23135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.113228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55112)
0928 21:16:41.113393 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.113407 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:41.113467 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:41.113907 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:41.113910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:41.114084 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:41.114207 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.114218 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.115257 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.115262 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.116103 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.116110 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.116694 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.116697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.116775 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.117502 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:41.117515 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:41.119329 (+ 1814us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:41.121373 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.121378 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.121380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.121655 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.121974 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.121980 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.121984 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.122082 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.122490 (+ 408us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:41.122493 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:41.122582 (+ 89us) client_negotiation.cc:769] Sending connection context
0928 21:16:41.122669 (+ 87us) client_negotiation.cc:240] Negotiation successful
0928 21:16:41.122791 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260928 21:16:41.123456 23287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.113329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55112 (local address 127.0.98.60:45593)
0928 21:16:41.113656 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.113660 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.113674 (+ 14us) server_negotiation.cc:407] Connection header received
0928 21:16:41.113713 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.113718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.113771 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.113860 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.114305 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.115087 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.116257 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.116569 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.117358 (+ 789us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.119447 (+ 2089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.119450 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.119452 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.119476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.121258 (+ 1782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.121787 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.121790 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.121792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.121866 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.122196 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.122199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.122201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.122385 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.123146 (+ 761us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.123219 (+ 73us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.123325 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247}
I20260928 21:16:41.123473 23284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } has no permanent_uuid. Determining permanent_uuid...
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260928 21:16:41.127190 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: Bootstrap starting.
I20260928 21:16:41.130219 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:41.130247 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.119707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55126)
0928 21:16:41.119815 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.119831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:41.119906 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:41.120789 (+ 883us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:41.120792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:41.121005 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:41.121152 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.121163 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.122418 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.122423 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.123224 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.123234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.124391 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.124398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.124492 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.125034 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:41.125054 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:41.126426 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:41.128509 (+ 2083us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.128515 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.128520 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.128782 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.129155 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.129158 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.129160 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.129248 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.129698 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:41.129701 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:41.129798 (+ 97us) client_negotiation.cc:769] Sending connection context
0928 21:16:41.129917 (+ 119us) client_negotiation.cc:240] Negotiation successful
0928 21:16:41.130061 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":2}
I20260928 21:16:41.130263 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.119969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.0.98.60:45593)
0928 21:16:41.120540 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.120549 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.120561 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:41.120596 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.120599 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.120661 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.121267 (+ 606us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.121354 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.122256 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.123902 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.124256 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.124749 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.126581 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.126584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.126587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.126611 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.128374 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.128940 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.128946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.128951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.129041 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.129368 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.129373 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.129378 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.129616 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.129710 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.129980 (+ 270us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.130095 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":228,"threads_started":1}
I20260928 21:16:41.131174 23133 log.cc:824] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:41.131465 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.120234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56688 (local address 127.0.98.61:32825)
0928 21:16:41.120342 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.120346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.120721 (+ 375us) server_negotiation.cc:407] Connection header received
0928 21:16:41.120944 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.120948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.120999 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.121055 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.122033 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.122791 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.123937 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.124276 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.124348 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.127673 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.127678 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.127680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.127714 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.129524 (+ 1810us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.129999 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.130002 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.130004 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.130091 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.130442 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.130447 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.130450 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.130655 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.130825 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.131212 (+ 387us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.131325 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":39,"server-negotiator.queue_time_us":39}
I20260928 21:16:41.133545 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: No bootstrap required, opened a new log
I20260928 21:16:41.133566 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: Bootstrap starting.
I20260928 21:16:41.135303 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
Sep 28 21:16:41 dist-test-slave-v7xz krb5kdc[23052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790630201, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260928 21:16:41.136445 23208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:41.136705 23107 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "91b36825dbaa44769aa2e0adca43097e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57562
I20260928 21:16:41.137800 23208 log.cc:824] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:41.137864 23133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.138104 23133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91b36825dbaa44769aa2e0adca43097e, State: Initialized, Role: FOLLOWER
I20260928 21:16:41.138451 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.139384 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.139524 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:41.139879 23133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:41.139928 23107 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260928 21:16:41.140025 23107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:41.140153 23208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: No bootstrap required, opened a new log
W20260928 21:16:41.141729 23086 tablet.cc:2369] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260928 21:16:41.141986 23107 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.143496 23107 leader_election.cc:288] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [CANDIDATE]: Term 1 election: Requested vote from peers 6cb00c78961a4afdbefee5330d09c7a7 (127.0.98.61:32825), de6345f5ba4646c7bde2546e6d2fe84c (127.0.98.60:45593)
I20260928 21:16:41.144352 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.124251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55132 (local address 127.0.98.60:45593)
0928 21:16:41.130942 (+ 6691us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.130947 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.130958 (+ 11us) server_negotiation.cc:407] Connection header received
0928 21:16:41.130990 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.130994 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.131038 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.131107 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.131578 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.132282 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.133487 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.133814 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.133913 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.136146 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.136149 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.136152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.136174 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.137929 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.138440 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.138445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.138448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.138528 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.138818 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.138821 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.138822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.139086 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.139186 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.144036 (+ 4850us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.144184 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6624}
I20260928 21:16:41.144937 23288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.123977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:45593 (local address 127.0.0.1:55132)
0928 21:16:41.124366 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0928 21:16:41.124379 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0928 21:16:41.124439 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0928 21:16:41.131197 (+ 6758us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0928 21:16:41.131199 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0928 21:16:41.131361 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0928 21:16:41.131453 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.131463 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.132398 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.132402 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.133362 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0928 21:16:41.133372 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.134011 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.134014 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0928 21:16:41.134083 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.134537 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0928 21:16:41.134550 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0928 21:16:41.136015 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0928 21:16:41.138042 (+ 2027us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.138046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.138048 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.138295 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.138617 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0928 21:16:41.138620 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0928 21:16:41.138622 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0928 21:16:41.138721 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0928 21:16:41.143554 (+ 4833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0928 21:16:41.143558 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0928 21:16:41.143758 (+ 200us) client_negotiation.cc:769] Sending connection context
0928 21:16:41.144524 (+ 766us) client_negotiation.cc:240] Negotiation successful
0928 21:16:41.144730 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313}
W20260928 21:16:41.145972 23304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:41.146041 23304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:41.146322 23208 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.147159 23208 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cb00c78961a4afdbefee5330d09c7a7, State: Initialized, Role: FOLLOWER
I20260928 21:16:41.147503 23208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.148064 23310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.148919 23208 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:41.151312 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: Bootstrap starting.
I20260928 21:16:41.148325 23065 master_runner.cc:186] Error getting master registration for 127.0.98.61:32825: OK, instance_id { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" instance_seqno: 1790630200865051 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:16:41.151844 23065 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:16:41.153060 23182 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91b36825dbaa44769aa2e0adca43097e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "6cb00c78961a4afdbefee5330d09c7a7"
I20260928 21:16:41.153318 23182 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:41.154306 23284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: Neither blocks nor log segments found. Creating new log.
I20260928 21:16:41.154745 23182 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b36825dbaa44769aa2e0adca43097e in term 1.
I20260928 21:16:41.155022 23310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:41.155282 23284 log.cc:824] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: Log is configured to *not* fsync() on all Append() calls
I20260928 21:16:41.155840 23081 leader_election.cc:302] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6cb00c78961a4afdbefee5330d09c7a7, 91b36825dbaa44769aa2e0adca43097e; no voters:
I20260928 21:16:41.156150 23293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 1 FOLLOWER]: Leader election won for term 1
I20260928 21:16:41.156359 23293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [term 1 LEADER]: Becoming Leader. State: Replica: 91b36825dbaa44769aa2e0adca43097e, State: Running, Role: LEADER
I20260928 21:16:41.156702 23293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.158540 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b36825dbaa44769aa2e0adca43097e. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.158672 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:41.161370 23140 master_runner.cc:186] Error getting master registration for 127.0.98.60:45593: OK, instance_id { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" instance_seqno: 1790630201085174 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260928 21:16:41.165243 23140 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260928 21:16:41.165664 23328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260928 21:16:41.165896 23323 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:41.165978 23323 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:41.166640 23328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260928 21:16:41.168141 23284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: No bootstrap required, opened a new log
I20260928 21:16:41.172510 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.153027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.0.98.60:45593)
0928 21:16:41.153166 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.153171 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.155566 (+ 2395us) server_negotiation.cc:407] Connection header received
0928 21:16:41.155850 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.155854 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.155909 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.155983 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.157609 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.158328 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.160007 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.160383 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.160460 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.163351 (+ 2891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.163355 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.163358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.163386 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.165162 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.169397 (+ 4235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.169401 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.169403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.169487 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.170509 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.170513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.170514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.170754 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.170868 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.172236 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.172361 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260928 21:16:41.172730 23284 raft_consensus.cc:348] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.173038 23284 raft_consensus.cc:729] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de6345f5ba4646c7bde2546e6d2fe84c, State: Initialized, Role: FOLLOWER
I20260928 21:16:41.173424 23284 consensus_queue.cc:260] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } }
I20260928 21:16:41.173628 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.149833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57564 (local address 127.0.98.62:44565)
0928 21:16:41.149944 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.149950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.156043 (+ 6093us) server_negotiation.cc:407] Connection header received
0928 21:16:41.156212 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.156216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.156265 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.156322 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.157329 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.158199 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.159422 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.159810 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.159903 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.167170 (+ 7267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.167174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.167177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.167203 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.168884 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.170073 (+ 1189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.170077 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.170078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.170161 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.171419 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.171423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.171425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.171652 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.171762 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.173356 (+ 1594us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.173472 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260928 21:16:41.174919 23284 sys_catalog.cc:564] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: configured and running, proceeding with master startup.
I20260928 21:16:41.175241 23258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "91b36825dbaa44769aa2e0adca43097e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "de6345f5ba4646c7bde2546e6d2fe84c"
I20260928 21:16:41.175484 23258 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [term 0 FOLLOWER]: Advancing to term 1
I20260928 21:16:41.177029 23258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 91b36825dbaa44769aa2e0adca43097e in term 1.
I20260928 21:16:41.178594 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.178731 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:41.181789 23213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:41.155064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56692 (local address 127.0.98.61:32825)
0928 21:16:41.155188 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.155193 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.170760 (+ 15567us) server_negotiation.cc:407] Connection header received
0928 21:16:41.170931 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.170935 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.170981 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.171030 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.171879 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.172662 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.173871 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:41.174221 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:41.174293 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:41.178138 (+ 3845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0928 21:16:41.178142 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0928 21:16:41.178145 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0928 21:16:41.178176 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0928 21:16:41.179914 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.180436 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.180440 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.180442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.180523 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0928 21:16:41.180804 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0928 21:16:41.180807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0928 21:16:41.180809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0928 21:16:41.181045 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0928 21:16:41.181177 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:41.181503 (+ 326us) server_negotiation.cc:299] Negotiation successful
0928 21:16:41.181626 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260928 21:16:41.188915 23258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [term 1 FOLLOWER]: Refusing update from remote peer 91b36825dbaa44769aa2e0adca43097e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:41.195187 23326 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [LEADER]: Connected to new peer: Peer: permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:41.196703 23182 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [term 1 FOLLOWER]: Refusing update from remote peer 91b36825dbaa44769aa2e0adca43097e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260928 21:16:41.197299 23293 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260928 21:16:41.201318 23310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b36825dbaa44769aa2e0adca43097e. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.201431 23310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:41.202884 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91b36825dbaa44769aa2e0adca43097e. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.203075 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:41.210033 23310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.210173 23310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7 [sys.catalog]: This master's current role is: FOLLOWER
I20260928 21:16:41.210570 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.210688 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:41.211654 23330 mvcc.cc:205] Tried to move back new op lower bound from 7334421304023793664 to 7334421303941554176. Current Snapshot: MvccSnapshot[applied={T|T < 7334421304023793664}]
I20260928 21:16:41.212352 23345 mvcc.cc:205] Tried to move back new op lower bound from 7334421304023793664 to 7334421303941554176. Current Snapshot: MvccSnapshot[applied={T|T < 7334421304023793664}]
I20260928 21:16:41.212664 23326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.212783 23326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e [sys.catalog]: This master's current role is: LEADER
I20260928 21:16:41.214291 23328 catalog_manager.cc:1347] Generated new cluster ID: 75dec9bb6ec74fbca4105fce5193e724
I20260928 21:16:41.214350 23328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260928 21:16:41.214534 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "91b36825dbaa44769aa2e0adca43097e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91b36825dbaa44769aa2e0adca43097e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44565 } } peers { permanent_uuid: "6cb00c78961a4afdbefee5330d09c7a7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 32825 } } peers { permanent_uuid: "de6345f5ba4646c7bde2546e6d2fe84c" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 45593 } } }
I20260928 21:16:41.214741 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c [sys.catalog]: This master's current role is: FOLLOWER
W20260928 21:16:41.234979 23342 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260928 21:16:41.235087 23342 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260928 21:16:41.245276 23328 catalog_manager.cc:1370] Generated new certificate authority record
I20260928 21:16:41.246639 23328 catalog_manager.cc:1504] Loading token signing keys...
I20260928 21:16:41.276952 23328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91b36825dbaa44769aa2e0adca43097e: Generated new TSK 0
I20260928 21:16:41.280503 23328 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260928 21:16:41.327260 23290 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.324740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55156 (local address 127.0.98.60:45593)
0928 21:16:41.324890 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.324896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.326074 (+ 1178us) server_negotiation.cc:407] Connection header received
0928 21:16:41.326269 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.326273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.326323 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.326412 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.327117 (+ 705us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55156: BlockingRecv error: recv got EOF from 127.0.0.1:55156 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260928 21:16:41.328970 23289 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.325426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57566 (local address 127.0.98.62:44565)
0928 21:16:41.325551 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.325555 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.327990 (+ 2435us) server_negotiation.cc:407] Connection header received
0928 21:16:41.328165 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.328168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.328216 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.328266 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.328812 (+ 546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57566: BlockingRecv error: recv got EOF from 127.0.0.1:57566 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260928 21:16:41.330847 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.329627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56704 (local address 127.0.98.61:32825)
0928 21:16:41.329749 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.329753 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.329843 (+ 90us) server_negotiation.cc:407] Connection header received
0928 21:16:41.330139 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.330142 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.330193 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.330243 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.330729 (+ 486us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56704: BlockingRecv error: recv got EOF from 127.0.0.1:56704 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260928 21:16:41.338873 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.337347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.0.98.61:32825)
0928 21:16:41.337475 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.337480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.337710 (+ 230us) server_negotiation.cc:407] Connection header received
0928 21:16:41.337970 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.337975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.338066 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.338127 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.338752 (+ 625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56716: BlockingRecv error: recv got EOF from 127.0.0.1:56716 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260928 21:16:41.347738 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.345714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.0.98.61:32825)
0928 21:16:41.345854 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.345859 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.346592 (+ 733us) server_negotiation.cc:407] Connection header received
0928 21:16:41.346812 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.346815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.346873 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.346942 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.347606 (+ 664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56728: BlockingRecv error: recv got EOF from 127.0.0.1:56728 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260928 21:16:41.355073 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.353259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56732 (local address 127.0.98.61:32825)
0928 21:16:41.353376 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.353381 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.354026 (+ 645us) server_negotiation.cc:407] Connection header received
0928 21:16:41.354243 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.354246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.354303 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.354357 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.354950 (+ 593us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56732: BlockingRecv error: recv got EOF from 127.0.0.1:56732 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260928 21:16:41.364660 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.362803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.0.98.61:32825)
0928 21:16:41.362966 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.362971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.363534 (+ 563us) server_negotiation.cc:407] Connection header received
0928 21:16:41.363763 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.363766 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.363823 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.363923 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.364530 (+ 607us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56746: BlockingRecv error: recv got EOF from 127.0.0.1:56746 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260928 21:16:41.378026 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.375961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.98.61:32825)
0928 21:16:41.376122 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.376127 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.376794 (+ 667us) server_negotiation.cc:407] Connection header received
0928 21:16:41.377089 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.377093 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.377157 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.377232 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.377900 (+ 668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56752: BlockingRecv error: recv got EOF from 127.0.0.1:56752 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260928 21:16:41.398417 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.397017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56756 (local address 127.0.98.61:32825)
0928 21:16:41.397158 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.397163 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.397364 (+ 201us) server_negotiation.cc:407] Connection header received
0928 21:16:41.397506 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.397510 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.397577 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.397694 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.398278 (+ 584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56756: BlockingRecv error: recv got EOF from 127.0.0.1:56756 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260928 21:16:41.433748 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.432507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56766 (local address 127.0.98.61:32825)
0928 21:16:41.432685 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.432690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.432848 (+ 158us) server_negotiation.cc:407] Connection header received
0928 21:16:41.433014 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.433017 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.433074 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.433180 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.433623 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56766: BlockingRecv error: recv got EOF from 127.0.0.1:56766 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260928 21:16:41.501401 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.500118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56782 (local address 127.0.98.61:32825)
0928 21:16:41.500311 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.500316 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.500449 (+ 133us) server_negotiation.cc:407] Connection header received
0928 21:16:41.500640 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.500644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.500697 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.500794 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.501291 (+ 497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56782: BlockingRecv error: recv got EOF from 127.0.0.1:56782 (error 108)
Metrics: {"server-negotiator.queue_time_us":92}
W20260928 21:16:41.632871 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.631702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.0.98.61:32825)
0928 21:16:41.631884 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.631889 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.632023 (+ 134us) server_negotiation.cc:407] Connection header received
0928 21:16:41.632146 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.632148 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.632197 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.632271 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.632706 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56786: BlockingRecv error: recv got EOF from 127.0.0.1:56786 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260928 21:16:41.892345 23213 negotiation.cc:337] Failed RPC negotiation. Trace:
0928 21:16:41.891259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56792 (local address 127.0.98.61:32825)
0928 21:16:41.891394 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:41.891398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:41.891486 (+ 88us) server_negotiation.cc:407] Connection header received
0928 21:16:41.891608 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:41.891610 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:41.891660 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:41.891732 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0928 21:16:41.892203 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56792: BlockingRecv error: recv got EOF from 127.0.0.1:56792 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:16:42.167405 23323 catalog_manager.cc:1259] Loaded cluster ID: 75dec9bb6ec74fbca4105fce5193e724
I20260928 21:16:42.167490 23323 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: loading cluster ID for follower catalog manager: success
I20260928 21:16:42.169464 23323 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: acquiring CA information for follower catalog manager: success
I20260928 21:16:42.170701 23323 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6cb00c78961a4afdbefee5330d09c7a7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:16:42.236330 23342 catalog_manager.cc:1259] Loaded cluster ID: 75dec9bb6ec74fbca4105fce5193e724
I20260928 21:16:42.236441 23342 catalog_manager.cc:1552] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: loading cluster ID for follower catalog manager: success
I20260928 21:16:42.238185 23342 catalog_manager.cc:1574] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: acquiring CA information for follower catalog manager: success
I20260928 21:16:42.239360 23342 catalog_manager.cc:1602] T 00000000000000000000000000000000 P de6345f5ba4646c7bde2546e6d2fe84c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260928 21:16:42.410185 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:42.406566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56796 (local address 127.0.98.61:32825)
0928 21:16:42.406822 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:42.406825 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:42.406841 (+ 16us) server_negotiation.cc:407] Connection header received
0928 21:16:42.406913 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:42.406915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:42.407041 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:42.407150 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:42.407391 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.407908 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.408753 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.409072 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.409153 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:42.409417 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:42.409850 (+ 433us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:42.409942 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:42.410014 (+ 72us) server_negotiation.cc:299] Negotiation successful
0928 21:16:42.410063 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":114,"threads_started":1}
I20260928 21:16:42.416234 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:42.413042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.0.98.60:45593)
0928 21:16:42.413246 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:42.413250 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:42.413262 (+ 12us) server_negotiation.cc:407] Connection header received
0928 21:16:42.413332 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:42.413334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:42.413438 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:42.413542 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:42.413721 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.414214 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.414952 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.415161 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.415234 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:42.415485 (+ 251us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:42.415851 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:42.415937 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:42.416002 (+ 65us) server_negotiation.cc:299] Negotiation successful
0928 21:16:42.416045 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":57,"threads_started":1}
I20260928 21:16:42.423215 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:42.418965 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56806 (local address 127.0.98.61:32825)
0928 21:16:42.419158 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:42.419165 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:42.419324 (+ 159us) server_negotiation.cc:407] Connection header received
0928 21:16:42.419504 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:42.419507 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:42.419564 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:42.419638 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:42.420154 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.420956 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.422068 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.422389 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.422481 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:42.422731 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:42.422804 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:42.422883 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:42.423003 (+ 120us) server_negotiation.cc:299] Negotiation successful
0928 21:16:42.423054 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260928 21:16:42.423456 23421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:42.419369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55174 (local address 127.0.98.60:45593)
0928 21:16:42.419500 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:42.419504 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:42.419825 (+ 321us) server_negotiation.cc:407] Connection header received
0928 21:16:42.419966 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:42.419971 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:42.420019 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:42.420096 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:42.420310 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.421065 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.421969 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.422225 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.422332 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:42.423006 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:42.423078 (+ 72us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:42.423124 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:42.423284 (+ 160us) server_negotiation.cc:299] Negotiation successful
0928 21:16:42.423336 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260928 21:16:42.424014 23427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0928 21:16:42.418924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57582 (local address 127.0.98.62:44565)
0928 21:16:42.419323 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0928 21:16:42.419327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0928 21:16:42.419340 (+ 13us) server_negotiation.cc:407] Connection header received
0928 21:16:42.419429 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0928 21:16:42.419432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0928 21:16:42.419602 (+ 170us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0928 21:16:42.419748 (+ 146us) server_negotiation.cc:226] Negotiated authn=TOKEN
0928 21:16:42.420131 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.420725 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.421933 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0928 21:16:42.422278 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0928 21:16:42.422373 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0928 21:16:42.423058 (+ 685us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0928 21:16:42.423411 (+ 353us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0928 21:16:42.423509 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0928 21:16:42.423819 (+ 310us) server_negotiation.cc:299] Negotiation successful
0928 21:16:42.423879 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":72,"threads_started":1}
I20260928 21:16:42.427182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 23065
I20260928 21:16:42.433938 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 23140
I20260928 21:16:42.439888 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task8iL2mC/build/debug/bin/kudu with pid 23215
2026-09-28T21:16:42Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3139 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6324 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (286312 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST